Nov 02 13:42:58.118277 ip-10-0-131-98 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 13:42:58.532776 ip-10-0-131-98 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:42:58.532776 ip-10-0-131-98 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 13:42:58.532776 ip-10-0-131-98 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:42:58.532776 ip-10-0-131-98 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:42:58.537447 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537351    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:58.537447 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537446    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:58.537447 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537451    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:58.537447 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537454    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537457    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537460    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537463    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537466    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537471    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537476    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537479    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537482    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537484    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537487    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537490    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537493    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537495    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537498    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537500    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537504    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537506    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537509    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537511    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:58.537579 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537514    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537517    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537519    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537521    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537524    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537536    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537540    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537542    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537545    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537547    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537552    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537555    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537558    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537561    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537564    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537567    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537572    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537575    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537578    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:58.538069 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537581    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537584    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537587    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537590    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537593    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537596    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537599    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537602    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537605    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537607    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537610    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537612    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537616    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537620    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537622    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537624    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537627    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537629    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537632    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537635    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:58.538529 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537637    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537640    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537642    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537646    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537648    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537651    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537653    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537657    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537660    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537662    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537665    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537667    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537670    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537673    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537676    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537679    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537682    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537684    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537688    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:58.539021 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537690    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537693    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537695    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537698    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537700    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537703    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537705    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537708    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537710    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537712    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537715    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537717    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537720    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537723    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537726    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537728    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537731    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537733    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537737    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:58.539472 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537740    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537743    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537746    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537748    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537751    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.537753    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538577    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538586    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538591    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538595    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538598    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538601    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538604    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538607    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538610    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538612    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538617    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538620    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538623    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:58.540035 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538626    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538629    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538631    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538634    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538636    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538640    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538643    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538646    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538649    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538651    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538654    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538657    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538659    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538663    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538666    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538668    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538671    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538673    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538676    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:58.540512 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538679    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538682    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538684    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538687    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538690    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538692    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538695    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538697    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538700    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538703    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538706    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538708    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538711    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538713    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538716    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538718    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538721    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538724    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538726    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538729    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:58.541006 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538732    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538735    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538737    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538740    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538743    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538745    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538748    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538751    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538753    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538756    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538758    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538761    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538763    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538766    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538769    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538772    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538775    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538778    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538780    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538783    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:58.541485 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538785    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538788    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538791    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538793    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538796    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538798    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538800    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538803    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538805    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538808    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538812    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538814    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538817    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538820    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538822    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538825    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538827    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538830    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538832    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538835    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:58.541980 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538837    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538840    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538842    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538845    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538847    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538850    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538852    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538855    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538857    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538860    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538862    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538865    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538870    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.538872    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539420    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539426    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539430    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539433    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539436    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:58.542470 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539439    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539443    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539445    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539448    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539451    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539455    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539457    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539460    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539463    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539465    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539468    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539471    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539474    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539476    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539479    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539482    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539485    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539487    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539490    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:58.542957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539493    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539500    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539502    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539505    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539508    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539510    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539513    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539516    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539521    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539524    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539527    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539530    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539533    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539535    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539537    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539540    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539543    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539546    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539549    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539551    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:58.543419 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539554    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539556    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539559    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539562    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539564    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539567    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539569    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539572    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539574    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539577    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539580    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539582    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539585    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539589    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539592    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539594    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539597    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539599    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539602    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539605    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:58.543892 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539607    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539618    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539621    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539623    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539627    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539629    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539632    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539635    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539637    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539640    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539643    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539645    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539649    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539653    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539656    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539660    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539662    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539665    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539668    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:58.544389 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539671    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539674    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539677    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539680    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539682    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539685    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539689    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539691    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539694    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539696    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539699    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539701    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539705    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539707    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539710    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539718    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539721    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539724    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539726    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:58.544879 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539730    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539733    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539735    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.539738    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540781    2581 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540794    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540801    2581 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540807    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540812    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540816    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540821    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540825    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540829    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540832    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540836    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540840    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540843    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540846    2581 flags.go:64] FLAG: --cgroup-root=""
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540849    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540853    2581 flags.go:64] FLAG: --client-ca-file=""
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540856    2581 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540859    2581 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540869    2581 flags.go:64] FLAG: --cluster-domain=""
Nov 02 13:42:58.545391 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540873    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540876    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540879    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540883    2581 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540887    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540891    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540895    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540898    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540902    2581 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540905    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540909    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540912    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540915    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540920    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540923    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540926    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540929    2581 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540942    2581 flags.go:64] FLAG: --enable-server="true"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540945    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540953    2581 flags.go:64] FLAG: --event-burst="100"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540956    2581 flags.go:64] FLAG: --event-qps="50"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540960    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540963    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540966    2581 flags.go:64] FLAG: --eviction-hard=""
Nov 02 13:42:58.545949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540970    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540974    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540977    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540980    2581 flags.go:64] FLAG: --eviction-soft=""
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540984    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540987    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540990    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540993    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540996    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.540999    2581 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541002    2581 flags.go:64] FLAG: --feature-gates=""
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541009    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541012    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541016    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541019    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541026    2581 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541030    2581 flags.go:64] FLAG: --help="false"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541034    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-131-98.ec2.internal"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541037    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541040    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541043    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541046    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541050    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541053    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 13:42:58.546627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541056    2581 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541059    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541062    2581 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541065    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541069    2581 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541072    2581 flags.go:64] FLAG: --kube-reserved=""
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541074    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541077    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541080    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541083    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541086    2581 flags.go:64] FLAG: --lock-file=""
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541089    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541093    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541097    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541104    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541107    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541110    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541113    2581 flags.go:64] FLAG: --logging-format="text"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541116    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541120    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541122    2581 flags.go:64] FLAG: --manifest-url=""
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541125    2581 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541131    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541134    2581 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541144    2581 flags.go:64] FLAG: --max-pods="110"
Nov 02 13:42:58.547224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541148    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541150    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541154    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541157    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541160    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541164    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541167    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541176    2581 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541180    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541183    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541186    2581 flags.go:64] FLAG: --pod-cidr=""
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541189    2581 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541192    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541195    2581 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541198    2581 flags.go:64] FLAG: --port="10250"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541201    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541204    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c719521f980d5365"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541207    2581 flags.go:64] FLAG: --qos-reserved=""
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541210    2581 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541213    2581 flags.go:64] FLAG: --register-node="true"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541216    2581 flags.go:64] FLAG: --register-with-taints=""
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541220    2581 flags.go:64] FLAG: --registry-burst="10"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541223    2581 flags.go:64] FLAG: --registry-qps="5"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541226    2581 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541229    2581 flags.go:64] FLAG: --reserved-memory=""
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541233    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 13:42:58.547848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541236    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541239    2581 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541242    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541245    2581 flags.go:64] FLAG: --runonce="false"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541249    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541252    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541256    2581 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541259    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541262    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541265    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541269    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541272    2581 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541275    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541277    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541280    2581 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541283    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541288    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541291    2581 flags.go:64] FLAG: --system-cgroups=""
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541294    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541300    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541304    2581 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541307    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541312    2581 flags.go:64] FLAG: --tls-min-version=""
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541315    2581 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541318    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 13:42:58.548520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541321    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541324    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541328    2581 flags.go:64] FLAG: --v="2"
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541332    2581 flags.go:64] FLAG: --version="false"
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541337    2581 flags.go:64] FLAG: --vmodule=""
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541341    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541345    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541530    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541536    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541540    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541543    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541546    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541551    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541553    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541556    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541560    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541562    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541565    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541568    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541570    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541573    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:58.549178 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541576    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541578    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541583    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541586    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541588    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541591    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541594    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541597    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541599    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541602    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541604    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541607    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541610    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541612    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541615    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541617    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541620    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541622    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541625    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:58.549682 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541628    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541631    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541633    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541636    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541639    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541642    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541645    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541648    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541651    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541653    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541656    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541659    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541661    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541664    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541666    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541670    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541674    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541676    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541679    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541681    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:58.550203 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541684    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541687    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541689    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541692    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541695    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541697    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541700    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541702    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541705    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541707    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541710    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541713    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541716    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541719    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541723    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541727    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541731    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541734    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541736    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:58.550702 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541739    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541742    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541745    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541748    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541750    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541753    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541756    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541758    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541762    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541764    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541767    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541770    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541772    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541775    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541777    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541780    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541782    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541785    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541788    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541790    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:58.551171 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541793    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541795    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541798    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541800    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541803    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541805    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541808    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541811    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541813    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541816    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541819    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541822    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541824    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541827    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.541856    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541896    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:58.551633 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541901    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541904    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541907    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541909    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541912    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541916    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541919    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541921    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541924    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541927    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541930    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541946    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541950    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541952    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541955    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541958    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541961    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541963    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541966    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541969    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:58.552057 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541971    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541974    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541976    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541979    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541981    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541985    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541988    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541991    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541994    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.541997    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542000    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542003    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542008    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542012    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542015    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542018    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542020    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542024    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542027    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:58.552533 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542030    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542034    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542037    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542039    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542042    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542044    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542047    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542049    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542052    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542055    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542057    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542060    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542062    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542065    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542068    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542070    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542073    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542075    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542080    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542083    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:58.553023 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542103    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542107    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542111    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542114    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542116    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542120    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542122    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542125    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542127    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542130    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542134    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542137    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542140    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542144    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542147    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542151    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542153    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542156    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542159    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:58.553507 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542161    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542171    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542175    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542179    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542182    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542184    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542187    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542189    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542192    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542195    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542198    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542202    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542204    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542208    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542211    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542213    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542216    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542219    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542221    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:58.554100 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542225    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542227    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542230    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542233    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542237    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542240    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542242    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.542245    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.549195    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.549216    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.549232    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.549237    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549298    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549304    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549307    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549311    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549314    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549317    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549319    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549322    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:58.554558 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549326    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549328    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549331    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549334    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549336    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549339    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549342    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549345    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549348    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549350    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549353    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549356    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549359    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549361    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549363    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549366    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549368    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549371    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549374    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549377    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:58.555050 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549379    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549383    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549386    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549389    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549391    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549394    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549396    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549399    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549402    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549404    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549407    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549409    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549412    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549414    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549417    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549420    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549422    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549425    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549427    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:58.555516 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549430    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549432    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549435    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549438    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549440    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549443    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549445    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549448    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549451    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549453    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549456    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549458    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549461    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549464    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549467    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549470    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549473    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549476    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549478    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549481    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:58.555990 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549483    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549486    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549488    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549491    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549493    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549496    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549499    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549501    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549504    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549507    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549509    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549512    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549514    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549517    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549520    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549522    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549525    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549527    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549530    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549533    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:58.556466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549536    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549538    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549541    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549543    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549547    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549550    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549554    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549560    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549564    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549575    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549579    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549583    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549585    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549588    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549591    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549594    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549596    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549599    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:58.556953 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549601    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.549634    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549786    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549790    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549793    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549796    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549800    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549803    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549806    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549808    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549811    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549814    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549816    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549819    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549822    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549825    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:58.557403 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549827    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549830    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549833    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549835    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549838    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549841    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549843    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549847    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549850    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549852    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549855    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549858    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549860    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549863    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549866    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549868    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549871    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549873    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549876    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549879    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:58.557794 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549881    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549883    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549886    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549890    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549894    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549897    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549899    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549902    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549905    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549908    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549911    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549913    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549916    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549918    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549921    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549923    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549926    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549929    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549946    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:58.558661 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549950    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549956    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549960    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549964    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549967    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549970    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549973    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549975    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549978    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549980    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549983    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549986    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549988    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549991    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549994    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549997    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.549999    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550002    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550004    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550007    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:58.559377 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550010    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550013    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550017    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550019    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550022    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550025    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550027    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550030    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550032    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550035    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550037    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550040    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550042    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550045    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550048    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550050    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550053    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550055    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550058    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:58.559957 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550060    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550063    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550065    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550068    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550070    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550074    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550078    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550081    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550084    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550086    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550089    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550091    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550094    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550097    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.550122    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:42:58.560422 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550155    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550159    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550163    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550166    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550168    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550171    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550173    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550176    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550179    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550181    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550184    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550187    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550189    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550192    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550195    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550198    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550200    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550203    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550206    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550209    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:58.560788 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550211    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550214    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550217    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550220    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550222    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550225    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550227    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550230    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550232    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550235    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550238    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550240    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550242    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550245    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550248    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550250    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550252    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550255    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550259    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:58.561306 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550263    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550266    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550269    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550271    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550274    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550277    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550280    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550283    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550286    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550288    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550291    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550293    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550296    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550299    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550301    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550304    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550306    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550309    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550311    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:58.561759 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550314    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550316    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550319    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550321    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550324    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550327    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550329    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550332    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550335    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550338    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550340    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550343    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550346    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550348    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550351    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550353    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550355    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550358    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550360    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:58.562289 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550363    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550365    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550368    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550371    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550374    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550376    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550379    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550383    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550387    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550390    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550392    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550396    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550399    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550401    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550405    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550408    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550410    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550413    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550415    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:58.562772 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550418    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550420    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550423    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550425    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550428    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550431    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550433    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550436    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550439    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:58.550442    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.550506    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.552622    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.553773    2581 server.go:1014] "Starting client certificate rotation"
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.553871    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.553905    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.556374    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:42:58.563238 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.560185    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:42:58.567005 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.566983    2581 log.go:25] "Validated CRI v1 runtime API"
Nov 02 13:42:58.570358 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.570340    2581 log.go:25] "Validated CRI v1 image API"
Nov 02 13:42:58.571224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.571196    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 13:42:58.575144 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.575115    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 99574455-4c74-4072-ad76-6ccfdf42f2e1:/dev/nvme0n1p3 9eda1afd-a622-451f-94f2-aa91a5e7518e:/dev/nvme0n1p4]
Nov 02 13:42:58.575198 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.575144    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 13:42:58.580605 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.580471    2581 manager.go:217] Machine: {Timestamp:2026-11-02 13:42:58.578996989 +0000 UTC m=+0.342430594 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3117199 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cd6418663bef73cf6076002ca7a25 SystemUUID:ec2cd641-8663-bef7-3cf6-076002ca7a25 BootID:3d677981-2928-41b1-ab67-2bbd40e508da Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:41:bc:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:41:bc:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:4f:1d:7f:3a:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 13:42:58.580605 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.580600    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 13:42:58.580733 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.580693    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 13:42:58.583495 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.583461    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 13:42:58.583642 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.583496    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 13:42:58.583725 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.583644    2581 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 13:42:58.583725 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.583653    2581 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 13:42:58.583725 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.583693    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:42:58.583725 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.583711    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:42:58.584568 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.584555    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 13:42:58.588694 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.588680    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 13:42:58.588845 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.588835    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 13:42:58.589754 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.589742    2581 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 13:42:58.589795 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.589762    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 13:42:58.589795 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.589782    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 13:42:58.589845 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.589798    2581 kubelet.go:405] "Adding apiserver pod source"
Nov 02 13:42:58.589845 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.589809    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 13:42:58.590254 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.590239    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:42:58.593530 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.593508    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 13:42:58.599607 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.599588    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:42:58.600351 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.600338    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 13:42:58.600393 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.600369    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 13:42:58.600538 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.600528    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 13:42:58.600595 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.600544    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 13:42:58.600595 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.600551    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 13:42:58.600595 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.600557    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 13:42:58.600595 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.600563    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 13:42:58.600595 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.600570    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 13:42:58.601016 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.601006    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 13:42:58.601052 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.601017    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 13:42:58.601052 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.601024    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 13:42:58.601052 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.601031    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 13:42:58.601052 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.601040    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 13:42:58.601052 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.601050    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 13:42:58.601509 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.601501    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 13:42:58.601539 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.601511    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 13:42:58.606539 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.606517    2581 server.go:1263] "Started kubelet"
Nov 02 13:42:58.606656 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.606588    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 13:42:58.606746 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.606675    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 13:42:58.606873 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.606767    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 13:42:58.607532 ip-10-0-131-98 systemd[1]: Started Kubernetes Kubelet.
Nov 02 13:42:58.607899 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.607880    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 13:42:58.610621 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.610606    2581 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 13:42:58.613645 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.613619    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 13:42:58.614143 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.614127    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 13:42:58.614143 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.614139    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 13:42:58.614273 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.614224    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:42:58.614819 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.614802    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 13:42:58.614819 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.614802    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 13:42:58.614948 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.614825    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 13:42:58.614948 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.614893    2581 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 13:42:58.614948 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.614902    2581 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 13:42:58.615302 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.615284    2581 factory.go:55] Registering systemd factory
Nov 02 13:42:58.615302 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.615302    2581 factory.go:223] Registration of the systemd container factory successfully
Nov 02 13:42:58.615599 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.615582    2581 factory.go:153] Registering CRI-O factory
Nov 02 13:42:58.615647 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.615603    2581 factory.go:223] Registration of the crio container factory successfully
Nov 02 13:42:58.615680 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.615660    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 13:42:58.615709 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.615681    2581 factory.go:103] Registering Raw factory
Nov 02 13:42:58.615709 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.615697    2581 manager.go:1199] Started watching for new ooms in manager
Nov 02 13:42:58.616632 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.616618    2581 manager.go:319] Starting recovery of all containers
Nov 02 13:42:58.621946 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.621895    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 13:42:58.630887 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.630859    2581 manager.go:324] Recovery completed
Nov 02 13:42:58.635596 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.635455    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:58.637453 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.637427    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:58.637544 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.637504    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:58.637544 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.637521    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:58.638132 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.638110    2581 cpu_manager.go:226] "Starting" policy="none"
Nov 02 13:42:58.638132 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.638129    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:58.638257 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.638128    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 13:42:58.638257 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.638232    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 13:42:58.638993 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.638973    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:58.639074 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.639007    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:58.639074 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.639022    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:58.642778 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.642758    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:42:58.643064 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.643040    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 13:42:58.644086 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.643083    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be765620c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.60647374 +0000 UTC m=+0.369907345,LastTimestamp:2026-11-02 13:42:58.60647374 +0000 UTC m=+0.369907345,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:58.660151 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.660116    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 13:42:58.660304 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.660177    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 13:42:58.660304 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.660194    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 13:42:58.660304 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.660163    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93e1627  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637452839 +0000 UTC m=+0.400886451,LastTimestamp:2026-11-02 13:42:58.637452839 +0000 UTC m=+0.400886451,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:58.687803 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.687701    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93efd2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637511981 +0000 UTC m=+0.400945596,LastTimestamp:2026-11-02 13:42:58.637511981 +0000 UTC m=+0.400945596,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:58.714730 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.714700    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:42:58.723197 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.723110    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93f3706  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.63752679 +0000 UTC m=+0.400960399,LastTimestamp:2026-11-02 13:42:58.63752679 +0000 UTC m=+0.400960399,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:58.740577 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.740484    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93e1627\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93e1627  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637452839 +0000 UTC m=+0.400886451,LastTimestamp:2026-11-02 13:42:58.638991534 +0000 UTC m=+0.402425146,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:58.764162 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.764064    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93efd2d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93efd2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637511981 +0000 UTC m=+0.400945596,LastTimestamp:2026-11-02 13:42:58.639013739 +0000 UTC m=+0.402447350,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:58.776111 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.776082    2581 policy_none.go:50] "Start"
Nov 02 13:42:58.776111 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.776110    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 13:42:58.776281 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.776127    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 13:42:58.778347 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.778329    2581 policy_none.go:44] "Start"
Nov 02 13:42:58.802667 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.790391    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93f3706\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93f3706  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.63752679 +0000 UTC m=+0.400960399,LastTimestamp:2026-11-02 13:42:58.639027463 +0000 UTC m=+0.402461077,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:58.814989 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.814806    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:42:58.828673 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.828640    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 13:42:58.834308 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.834278    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 13:42:58.834308 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.834308    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 13:42:58.834504 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.834331    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 13:42:58.834504 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.834373    2581 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 13:42:58.846003 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.845977    2581 manager.go:341] "Starting Device Plugin manager"
Nov 02 13:42:58.846139 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.846013    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 13:42:58.846139 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.846023    2581 server.go:87] "Starting device plugin registration server"
Nov 02 13:42:58.846282 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.846272    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 13:42:58.846314 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.846283    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 13:42:58.846420 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.846398    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 13:42:58.846539 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.846480    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 13:42:58.846539 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.846488    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 13:42:58.847097 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.847073    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 13:42:58.847190 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.847134    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:42:58.896053 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.896018    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Nov 02 13:42:58.896053 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.896037    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 02 13:42:58.896274 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.896068    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3bf5864016  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.843508758 +0000 UTC m=+0.606942350,LastTimestamp:2026-11-02 13:42:58.843508758 +0000 UTC m=+0.606942350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:58.934908 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.934868    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"]
Nov 02 13:42:58.935104 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.934969    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:58.936024 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.936008    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:58.936120 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.936043    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:58.936120 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.936057    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:58.938299 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.938281    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:58.938453 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.938425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:58.938506 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.938469    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:58.939111 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.939094    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:58.939194 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.939124    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:58.939194 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.939133    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:58.939194 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.939094    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:58.939333 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.939215    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:58.939333 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.939238    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:58.941314 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.941297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:58.941402 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.941330    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:58.942170 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.942148    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:58.942247 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.942182    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:58.942247 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.942193    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:58.944966 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.944878    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3bf5ab70d5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.845946069 +0000 UTC m=+0.609379676,LastTimestamp:2026-11-02 13:42:58.845946069 +0000 UTC m=+0.609379676,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:58.946696 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.946680    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:58.947462 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.947448    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:58.947532 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.947476    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:58.947532 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.947485    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:58.947532 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:58.947507    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:42:58.966068 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.966046    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-98.ec2.internal\" not found" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:42:58.969144 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:58.969123    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-98.ec2.internal\" not found" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.003033 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.002901    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93e1627\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93e1627  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637452839 +0000 UTC m=+0.400886451,LastTimestamp:2026-11-02 13:42:58.936024543 +0000 UTC m=+0.699458154,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.016786 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.016748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.016873 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.016791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.016873 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.016811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41b5d311172faec6b74eed63c4c5b623-config\") pod \"kube-apiserver-proxy-ip-10-0-131-98.ec2.internal\" (UID: \"41b5d311172faec6b74eed63c4c5b623\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.062217 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.062134    2581 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.062350 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.062171    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93efd2d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93efd2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637511981 +0000 UTC m=+0.400945596,LastTimestamp:2026-11-02 13:42:58.936049632 +0000 UTC m=+0.699483238,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.116928 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.116893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.116928 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.116927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41b5d311172faec6b74eed63c4c5b623-config\") pod \"kube-apiserver-proxy-ip-10-0-131-98.ec2.internal\" (UID: \"41b5d311172faec6b74eed63c4c5b623\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.117181 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.116967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.117181 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.117018    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.117181 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.117022    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.117181 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.117018    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41b5d311172faec6b74eed63c4c5b623-config\") pod \"kube-apiserver-proxy-ip-10-0-131-98.ec2.internal\" (UID: \"41b5d311172faec6b74eed63c4c5b623\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.117820 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.117731    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93f3706\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93f3706  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.63752679 +0000 UTC m=+0.400960399,LastTimestamp:2026-11-02 13:42:58.936063421 +0000 UTC m=+0.699497031,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.165184 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.165095    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93e1627\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93e1627  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637452839 +0000 UTC m=+0.400886451,LastTimestamp:2026-11-02 13:42:58.939113666 +0000 UTC m=+0.702547271,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.218502 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.217230    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93efd2d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93efd2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637511981 +0000 UTC m=+0.400945596,LastTimestamp:2026-11-02 13:42:58.939128529 +0000 UTC m=+0.702562134,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.262720 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.262683    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:59.263683 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.263664    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:59.263782 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.263698    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:59.263782 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.263714    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:59.263782 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.263756    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.268868 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.268851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.270685 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.270595    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93f3706\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93f3706  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.63752679 +0000 UTC m=+0.400960399,LastTimestamp:2026-11-02 13:42:58.939137661 +0000 UTC m=+0.702571270,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.273316 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.273297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.317039 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.316928    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms"
Nov 02 13:42:59.317207 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.316938    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93e1627\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93e1627  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637452839 +0000 UTC m=+0.400886451,LastTimestamp:2026-11-02 13:42:58.939195147 +0000 UTC m=+0.702628755,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.335595 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.335563    2581 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.335771 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.335584    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93efd2d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93efd2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637511981 +0000 UTC m=+0.400945596,LastTimestamp:2026-11-02 13:42:58.939224695 +0000 UTC m=+0.702658304,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.371736 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.371634    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93f3706\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93f3706  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.63752679 +0000 UTC m=+0.400960399,LastTimestamp:2026-11-02 13:42:58.939244569 +0000 UTC m=+0.702678175,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.429017 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.428889    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93e1627\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93e1627  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637452839 +0000 UTC m=+0.400886451,LastTimestamp:2026-11-02 13:42:58.942166537 +0000 UTC m=+0.705600143,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.485087 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.484963    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93efd2d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93efd2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637511981 +0000 UTC m=+0.400945596,LastTimestamp:2026-11-02 13:42:58.942186883 +0000 UTC m=+0.705620488,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.533233 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.533121    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93f3706\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93f3706  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.63752679 +0000 UTC m=+0.400960399,LastTimestamp:2026-11-02 13:42:58.942197084 +0000 UTC m=+0.705630688,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.590145 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.589992    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93e1627\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93e1627  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637452839 +0000 UTC m=+0.400886451,LastTimestamp:2026-11-02 13:42:58.947462367 +0000 UTC m=+0.710895971,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.651457 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.651328    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93efd2d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93efd2d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637511981 +0000 UTC m=+0.400945596,LastTimestamp:2026-11-02 13:42:58.947480488 +0000 UTC m=+0.710914095,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.699054 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.699018    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:42:59.699225 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.699002    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93f3706\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93f3706  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.63752679 +0000 UTC m=+0.400960399,LastTimestamp:2026-11-02 13:42:58.947489401 +0000 UTC m=+0.710923006,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.709037 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.708999    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:59.710235 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.710211    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:59.710397 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.710247    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:59.710397 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.710261    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:59.735711 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.735681    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:59.737456 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.737425    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:59.737456 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.737460    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:59.737608 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.737471    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:59.737608 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.737503    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.748184 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.748083    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-98.ec2.internal.18e43e3be93e1627\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-98.ec2.internal.18e43e3be93e1627  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-98.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:58.637452839 +0000 UTC m=+0.400886451,LastTimestamp:2026-11-02 13:42:59.263682413 +0000 UTC m=+1.027116017,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:42:59.845448 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.845368    2581 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:42:59.882610 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:59.882560    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b5d311172faec6b74eed63c4c5b623.slice/crio-741a99eacd95e3aecc7c4e7631b2e615e502928b6e58c8abe3d4c76e2a6cb379 WatchSource:0}: Error finding container 741a99eacd95e3aecc7c4e7631b2e615e502928b6e58c8abe3d4c76e2a6cb379: Status 404 returned error can't find the container with id 741a99eacd95e3aecc7c4e7631b2e615e502928b6e58c8abe3d4c76e2a6cb379
Nov 02 13:42:59.889978 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:42:59.889957    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:42:59.893046 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.893000    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 13:42:59.893141 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:42:59.893103    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd005c7e1a378e0971dc1c77f9a40f556.slice/crio-d2b0f73d19c2fa0514d6a23b654b88058c0e04a50f1af46d30f711cbbcc2277f WatchSource:0}: Error finding container d2b0f73d19c2fa0514d6a23b654b88058c0e04a50f1af46d30f711cbbcc2277f: Status 404 returned error can't find the container with id d2b0f73d19c2fa0514d6a23b654b88058c0e04a50f1af46d30f711cbbcc2277f
Nov 02 13:42:59.947781 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:42:59.947678    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-98.ec2.internal.18e43e3c33f67d72  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-98.ec2.internal,UID:41b5d311172faec6b74eed63c4c5b623,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\",Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:59.89105189 +0000 UTC m=+1.654485499,LastTimestamp:2026-11-02 13:42:59.89105189 +0000 UTC m=+1.654485499,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:43:00.004215 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:00.004116    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal.18e43e3c34459efc  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal,UID:d005c7e1a378e0971dc1c77f9a40f556,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\",Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:42:59.89623782 +0000 UTC m=+1.659671412,LastTimestamp:2026-11-02 13:42:59.89623782 +0000 UTC m=+1.659671412,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:43:00.170826 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:00.170719    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 13:43:00.170996 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:00.170836    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s"
Nov 02 13:43:00.283230 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:00.283192    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 02 13:43:00.283456 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:00.283259    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 13:43:00.646303 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:00.646265    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:00.647784 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:00.647747    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:00.647935 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:00.647791    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:00.647935 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:00.647808    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:00.647935 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:00.647850    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:00.658569 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:00.658539    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:43:00.768869 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:00.768826    2581 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:00.840746 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:00.840687    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal" event={"ID":"41b5d311172faec6b74eed63c4c5b623","Type":"ContainerStarted","Data":"741a99eacd95e3aecc7c4e7631b2e615e502928b6e58c8abe3d4c76e2a6cb379"}
Nov 02 13:43:00.842100 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:00.842070    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal" event={"ID":"d005c7e1a378e0971dc1c77f9a40f556","Type":"ContainerStarted","Data":"d2b0f73d19c2fa0514d6a23b654b88058c0e04a50f1af46d30f711cbbcc2277f"}
Nov 02 13:43:01.299980 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:01.298621    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal.18e43e3c84747b98  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal,UID:d005c7e1a378e0971dc1c77f9a40f556,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\" in 1.345s (1.345s including waiting). Image size: 272339155 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:43:01.241486232 +0000 UTC m=+3.004919851,LastTimestamp:2026-11-02 13:43:01.241486232 +0000 UTC m=+3.004919851,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:43:01.545607 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:01.545496    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-98.ec2.internal.18e43e3c9318dca5  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-98.ec2.internal,UID:41b5d311172faec6b74eed63c4c5b623,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\" in 1.596s (1.596s including waiting). Image size: 294084393 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:43:01.487140005 +0000 UTC m=+3.250573619,LastTimestamp:2026-11-02 13:43:01.487140005 +0000 UTC m=+3.250573619,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:43:01.595752 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:01.595641    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-98.ec2.internal.18e43e3c953db844  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-98.ec2.internal,UID:41b5d311172faec6b74eed63c4c5b623,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:43:01.523109956 +0000 UTC m=+3.286543571,LastTimestamp:2026-11-02 13:43:01.523109956 +0000 UTC m=+3.286543571,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:43:01.648685 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:01.648576    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal.18e43e3c95477f6c  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal,UID:d005c7e1a378e0971dc1c77f9a40f556,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:43:01.523750764 +0000 UTC m=+3.287184394,LastTimestamp:2026-11-02 13:43:01.523750764 +0000 UTC m=+3.287184394,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:43:01.698425 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.698383    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:43:01.698641 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:01.698348    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-98.ec2.internal.18e43e3c95c669c3  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-98.ec2.internal,UID:41b5d311172faec6b74eed63c4c5b623,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:43:01.532068291 +0000 UTC m=+3.295501911,LastTimestamp:2026-11-02 13:43:01.532068291 +0000 UTC m=+3.295501911,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:43:01.756250 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:01.756105    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal.18e43e3c95d74f23  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal,UID:d005c7e1a378e0971dc1c77f9a40f556,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-131-98.ec2.internal,},FirstTimestamp:2026-11-02 13:43:01.533175587 +0000 UTC m=+3.296609180,LastTimestamp:2026-11-02 13:43:01.533175587 +0000 UTC m=+3.296609180,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-98.ec2.internal,}"
Nov 02 13:43:01.819432 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:01.819393    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s"
Nov 02 13:43:01.846379 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.846331    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal" event={"ID":"d005c7e1a378e0971dc1c77f9a40f556","Type":"ContainerStarted","Data":"1f239c01336c55275fe87e7b32eeaab468bf85f53b8ce8f8b111b74cb0ed7410"}
Nov 02 13:43:01.846551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.846357    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:01.847982 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.847932    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal" event={"ID":"41b5d311172faec6b74eed63c4c5b623","Type":"ContainerStarted","Data":"a16f90d288e49eafb73084350bd6380bc2b4ab7f5fe64e1051f0d2e72fb277a4"}
Nov 02 13:43:01.848103 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.847961    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:01.848103 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.848007    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:01.848103 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.848023    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:01.848103 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.848037    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:01.848290 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:01.848216    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-98.ec2.internal\" not found" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:01.848890 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.848873    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:01.848955 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.848903    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:01.848955 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.848913    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:01.849091 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:01.849078    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-98.ec2.internal\" not found" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:01.957022 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.956975    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:01.958154 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.958134    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:01.958231 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.958169    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:01.958231 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:01.958185    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:02.370571 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.370526    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:02.373399 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.373380    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:02.373502 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.373418    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:02.373502 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.373430    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:02.373502 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.373462    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:02.480857 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:02.480818    2581 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:02.536276 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:02.536235    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 13:43:02.690173 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.690073    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:43:02.707253 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:02.707213    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 13:43:02.798162 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:02.798120    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 02 13:43:02.849448 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.849421    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:02.849448 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.849438    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:02.853588 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.853566    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:02.853588 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.853576    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:02.853724 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.853605    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:02.853724 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.853616    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:02.853724 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.853633    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:02.853724 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:02.853656    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:02.853901 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:02.853887    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-98.ec2.internal\" not found" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:02.854040 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:02.854008    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-98.ec2.internal\" not found" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:03.312037 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:03.311993    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 13:43:03.687583 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:03.687551    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:43:04.701059 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:04.701020    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:43:05.069391 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:05.069355    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s"
Nov 02 13:43:05.681114 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:05.681065    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:05.682374 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:05.682352    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:05.682452 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:05.682392    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:05.682452 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:05.682403    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:05.682452 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:05.682447    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:05.701759 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:05.701721    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:43:05.736477 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:05.736438    2581 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:06.123052 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:06.122999    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:06.124331 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:06.124310    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:06.124410 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:06.124352    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:06.124410 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:06.124367    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:06.699563 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:06.699531    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:43:06.902836 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:06.902794    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 13:43:06.952808 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:06.952722    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 13:43:07.697917 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:07.697879    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:43:07.912249 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:07.912209    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 02 13:43:08.317173 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:08.317135    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 13:43:08.654705 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:08.654612    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 13:43:08.657568 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:08.657541    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-psknk"
Nov 02 13:43:08.670925 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:08.670902    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-psknk"
Nov 02 13:43:08.713710 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:08.713678    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:08.742949 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:08.742920    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:08.817380 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:08.817351    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:08.848017 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:08.847985    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:09.109411 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.109384    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:09.109411 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:09.109415    2581 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:09.157738 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.157709    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:09.185455 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.185424    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:09.253463 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.253427    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:09.533358 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.533330    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:09.533358 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:09.533357    2581 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:09.553124 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.553088    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 13:43:09.675314 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.675256    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:38:08 +0000 UTC" deadline="2028-05-04 00:17:15.79425707 +0000 UTC"
Nov 02 13:43:09.675314 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.675302    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13162h34m6.118958879s"
Nov 02 13:43:09.836823 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.836737    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:09.864646 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.864619    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:09.934581 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:09.934554    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:10.207732 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:10.207647    2581 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:10.207732 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:10.207680    2581 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-131-98.ec2.internal" not found
Nov 02 13:43:11.502065 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:11.502030    2581 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-98.ec2.internal\" not found" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:12.137029 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.136833    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:12.137984 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.137966    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:12.138103 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.137995    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:12.138103 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.138005    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:12.138103 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.138032    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:12.161178 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.161156    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:12.161178 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:12.161179    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-98.ec2.internal\": node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:12.206674 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:12.206650    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:12.307677 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:12.307649    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:12.408262 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:12.408200    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:12.508685 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:12.508659    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:12.609645 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:12.609607    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:12.710496 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:12.710389    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:12.810969 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:12.810918    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:12.867383 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.867359    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal_d005c7e1a378e0971dc1c77f9a40f556/setup/0.log"
Nov 02 13:43:12.867498 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.867391    2581 generic.go:356] "Generic (PLEG): container finished" podID="d005c7e1a378e0971dc1c77f9a40f556" containerID="1f239c01336c55275fe87e7b32eeaab468bf85f53b8ce8f8b111b74cb0ed7410" exitCode=1
Nov 02 13:43:12.867498 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.867421    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal" event={"ID":"d005c7e1a378e0971dc1c77f9a40f556","Type":"ContainerDied","Data":"1f239c01336c55275fe87e7b32eeaab468bf85f53b8ce8f8b111b74cb0ed7410"}
Nov 02 13:43:12.867567 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.867544    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:12.868405 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.868391    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:12.868462 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.868418    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:12.868462 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:12.868429    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:12.868624 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:12.868612    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-98.ec2.internal\" not found" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:12.911901 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:12.911878    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:13.012654 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.012624    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:13.112872 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.112839    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:13.213853 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.213821    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:13.314738 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.314660    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:13.415107 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.415071    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:13.515523 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.515485    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:13.616508 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.616444    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:13.717081 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.717050    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:13.818058 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.818021    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:13.870471 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:13.870414    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal_d005c7e1a378e0971dc1c77f9a40f556/setup/0.log"
Nov 02 13:43:13.870471 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:13.870453    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal" event={"ID":"d005c7e1a378e0971dc1c77f9a40f556","Type":"ContainerStarted","Data":"065fc2231ae9c9877bb03017332062bf802feae90307084906dbfd48f2bb358d"}
Nov 02 13:43:13.870601 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:13.870555    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:13.871326 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:13.871312    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:13.871389 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:13.871339    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:13.871389 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:13.871351    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:13.871547 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.871536    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-98.ec2.internal\" not found" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:43:13.919005 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:13.918990    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:14.019971 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:14.019919    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:14.121039 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:14.120983    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:14.221408 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:14.221386    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:14.321943 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:14.321903    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:14.422399 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:14.422344    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:14.523271 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:14.523244    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:14.624202 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:14.624180    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:14.724758 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:14.724698    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:14.783251 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:14.783230    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:14.825440 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:14.825422    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:14.926378 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:14.926357    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:15.026896 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:15.026867    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:15.127694 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:15.127665    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:15.228121 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:15.228096    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:15.328687 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:15.328633    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:15.429069 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:15.429044    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:15.482241 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:15.482214    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:15.486910 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:15.486894    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:15.486990 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:15.486925    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:15.486990 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:15.486950    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:15.529836 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:15.529815    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:15.630284 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:15.630236    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:15.730770 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:15.730746    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:15.831156 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:15.831140    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:15.931897 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:15.931853    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:16.032341 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:16.032317    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:16.132962 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:16.132931    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:16.233368 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:16.233348    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:16.333876 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:16.333852    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:16.434290 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:16.434273    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:16.535311 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:16.535247    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:16.636242 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:16.636216    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:16.736753 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:16.736721    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:16.814404 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:16.814356    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:16.837658 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:16.837638    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:16.937756 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:16.937733    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:17.038681 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:17.038651    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:17.138967 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:17.138912    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:17.239413 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:17.239388    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:17.340337 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:17.340313    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:17.440910 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:17.440851    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:17.541306 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:17.541276    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:17.641673 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:17.641645    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:17.742109 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:17.742091    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:17.842766 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:17.842741    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:17.943700 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:17.943676    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:18.044181 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.044116    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:18.144924 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.144893    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:18.245332 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.245305    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:18.345871 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.345822    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:18.446291 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.446261    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:43:18.524995 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.524972    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:18.601049 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.600994    2581 apiserver.go:52] "Watching apiserver"
Nov 02 13:43:18.615279 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.615262    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:43:18.625012 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.624995    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 13:43:18.647302 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.647281    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:43:18.647373 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.647351    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:43:18.647410 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.647401    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 13:43:18.647722 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.647706    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-gkg2v","openshift-network-diagnostics/network-check-target-57d72","openshift-network-operator/iptables-alerter-4j587","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k","openshift-ovn-kubernetes/ovnkube-node-sr6pw","openshift-multus/multus-additional-cni-plugins-cgnfh","openshift-multus/network-metrics-daemon-jcqf2","openshift-dns/node-resolver-5jbwx","openshift-multus/multus-zf8tf","kube-system/global-pull-secret-syncer-vsx66","kube-system/konnectivity-agent-pd8nn","openshift-cluster-node-tuning-operator/tuned-xgnrc"]
Nov 02 13:43:18.652774 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.652757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.655678 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.655662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:18.655756 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.655724    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:18.658355 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.658340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.661001 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.660985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.664221 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.664205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.666954 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.666928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.669562 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.669544    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:18.669647 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.669608    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:18.669870 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.669857    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:43:18.671142 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.671114    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.671219 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.671143    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 13:43:18.671219 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.671160    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.671322 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.671284    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6lw86\""
Nov 02 13:43:18.672133 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.672115    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:18.674736 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.674718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.677809 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.677795    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:18.677888 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.677855    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:18.681263 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.681242    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:18.683667 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.683647    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 13:43:18.684067 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.684054    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"]
Nov 02 13:43:18.684126 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.684078    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"]
Nov 02 13:43:18.684170 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.684140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.687515 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.687497    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wp5qr"
Nov 02 13:43:18.700027 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.700009    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wp5qr"
Nov 02 13:43:18.716539 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.716519    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 13:43:18.771325 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771309    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.771422 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771328    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.771422 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771352    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 13:43:18.771638 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771603    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 13:43:18.771714 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771653    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.771714 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771653    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.771877 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771862    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 13:43:18.771993 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771927    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.771993 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771946    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 13:43:18.771993 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771948    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 13:43:18.771993 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771981    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.771993 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.771981    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 13:43:18.772290 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.772076    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 13:43:18.772290 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.772087    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.772290 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.772091    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4wcws\""
Nov 02 13:43:18.772290 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.772095    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.772290 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.772125    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.772290 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.772136    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qnz8k\""
Nov 02 13:43:18.772290 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.772136    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bqvhh\""
Nov 02 13:43:18.772290 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.772142    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.785165 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785146    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 13:43:18.785165 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785145    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 13:43:18.785242 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785170    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5dqzs\""
Nov 02 13:43:18.785411 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785400    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.785449 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785417    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 13:43:18.785449 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785427    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-52vg6\""
Nov 02 13:43:18.785449 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785431    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 13:43:18.785449 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785445    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 13:43:18.785570 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785454    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w4l5x\""
Nov 02 13:43:18.785570 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785474    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lvdl4\""
Nov 02 13:43:18.785570 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785462    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.785570 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.785465    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-47zgd\""
Nov 02 13:43:18.805605 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.805667 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9jp\" (UniqueName: \"kubernetes.io/projected/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-kube-api-access-cs9jp\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:18.805667 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.805667 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-kubelet\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.805766 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805709    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-daemon-config\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.805766 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-etc-kubernetes\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.805766 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.805860 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c640c634-2704-48e7-a121-e0975ae43e23-iptables-alerter-script\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.805860 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-netd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.805860 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-env-overrides\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.805965 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nfr8\" (UniqueName: \"kubernetes.io/projected/094d4ae0-f6f8-44ae-a658-03f6322a3223-kube-api-access-8nfr8\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.805965 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-bin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.805965 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-tuned\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.806065 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.805995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99d52cb6-4109-40d5-8fc7-fb6679a03d60-serviceca\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.806065 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-system-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.806065 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:18.806161 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-registration-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.806161 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-k8s-cni-cncf-io\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.806161 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72ca4634-5648-402b-aaef-13f8d4ee2b4b-agent-certs\") pod \"konnectivity-agent-pd8nn\" (UID: \"72ca4634-5648-402b-aaef-13f8d4ee2b4b\") " pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:18.806161 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysconfig\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.806279 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-lib-modules\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.806279 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdgft\" (UniqueName: \"kubernetes.io/projected/25220e90-f330-4bd6-b2ff-45eb20b344c2-kube-api-access-wdgft\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.806279 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-netns\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.806279 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-systemd\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.806279 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-systemd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.806434 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.806434 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-tmp-dir\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:18.806434 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-dbus\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:18.806434 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6b5\" (UniqueName: \"kubernetes.io/projected/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-kube-api-access-6q6b5\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.806434 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:18.806434 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-ovn\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.806434 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806406    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cnibin\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.806434 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-var-lib-kubelet\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-tmp\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-slash\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cni-binary-copy\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtmnw\" (UniqueName: \"kubernetes.io/projected/99d52cb6-4109-40d5-8fc7-fb6679a03d60-kube-api-access-mtmnw\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-etc-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-bin\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/528f5273-2e64-4225-b59b-1616674a15fa-ovn-node-metrics-cert\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-socket-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-device-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.806675 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbffg\" (UniqueName: \"kubernetes.io/projected/cff7b24a-4b98-4592-9344-5ec4930c6079-kube-api-access-jbffg\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-multus-certs\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72ca4634-5648-402b-aaef-13f8d4ee2b4b-konnectivity-ca\") pod \"konnectivity-agent-pd8nn\" (UID: \"72ca4634-5648-402b-aaef-13f8d4ee2b4b\") " pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-conf\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d52cb6-4109-40d5-8fc7-fb6679a03d60-host\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-log-socket\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cnibin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c640c634-2704-48e7-a121-e0975ae43e23-host-slash\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-node-log\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.806984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmv6\" (UniqueName: \"kubernetes.io/projected/528f5273-2e64-4225-b59b-1616674a15fa-kube-api-access-xrmv6\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-hosts-file\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.807071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-sys-fs\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsdb\" (UniqueName: \"kubernetes.io/projected/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-kube-api-access-vdsdb\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-systemd-units\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-var-lib-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-script-lib\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-socket-dir-parent\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-netns\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-run\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-conf-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qcz\" (UniqueName: \"kubernetes.io/projected/c640c634-2704-48e7-a121-e0975ae43e23-kube-api-access-v9qcz\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-config\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-system-cni-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-multus\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-modprobe-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.807551 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-host\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.808007 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-kubelet-config\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:18.808007 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-hostroot\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.808007 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-kubernetes\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.808007 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-sys\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.808007 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.807522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-kubelet\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.908092 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:18.908092 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cnibin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.908223 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.908223 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c640c634-2704-48e7-a121-e0975ae43e23-host-slash\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.908223 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-node-log\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.908223 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmv6\" (UniqueName: \"kubernetes.io/projected/528f5273-2e64-4225-b59b-1616674a15fa-kube-api-access-xrmv6\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.908223 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-hosts-file\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:18.908223 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.908178    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:18.908422 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908217    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.908422 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908236    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c640c634-2704-48e7-a121-e0975ae43e23-host-slash\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.908422 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908234    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-hosts-file\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:18.908422 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.908256    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs podName:cff7b24a-4b98-4592-9344-5ec4930c6079 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.408220408 +0000 UTC m=+21.171654017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs") pod "network-metrics-daemon-jcqf2" (UID: "cff7b24a-4b98-4592-9344-5ec4930c6079") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:18.908422 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908247    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cnibin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.908422 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908294    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-node-log\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.908422 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908383    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.908704 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.908736 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-sys-fs\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.908777 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsdb\" (UniqueName: \"kubernetes.io/projected/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-kube-api-access-vdsdb\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.908777 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-systemd-units\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.908777 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-var-lib-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.908889 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-script-lib\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.908889 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-socket-dir-parent\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.908889 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908827    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-sys-fs\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.908889 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-netns\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.908889 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908851    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-var-lib-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.908889 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-run\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.908889 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-conf-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908885    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-systemd-units\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qcz\" (UniqueName: \"kubernetes.io/projected/c640c634-2704-48e7-a121-e0975ae43e23-kube-api-access-v9qcz\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908895    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-socket-dir-parent\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-config\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908956    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-run\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.908978    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-conf-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-system-cni-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909043    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-multus\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909123    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-system-cni-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-modprobe-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909190    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-multus\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.909217 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-host\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-kubelet-config\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909278    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-modprobe-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-hostroot\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-kubernetes\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909327    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-kubelet-config\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-sys\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909339    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-host\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909354    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-netns\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-kubelet\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909404    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-sys\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9jp\" (UniqueName: \"kubernetes.io/projected/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-kube-api-access-cs9jp\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909415    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-kubelet\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909434    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-hostroot\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909452    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-kubernetes\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-kubelet\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.909730 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-daemon-config\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-etc-kubernetes\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c640c634-2704-48e7-a121-e0975ae43e23-iptables-alerter-script\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-netd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909673    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-kubelet\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-env-overrides\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909722    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-netd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909726    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-etc-kubernetes\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909749    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909794    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfr8\" (UniqueName: \"kubernetes.io/projected/094d4ae0-f6f8-44ae-a658-03f6322a3223-kube-api-access-8nfr8\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-bin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.909980    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-bin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-tuned\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99d52cb6-4109-40d5-8fc7-fb6679a03d60-serviceca\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-system-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.910369 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910218    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-config\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910222    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-registration-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.910398    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910407    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-system-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-k8s-cni-cncf-io\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72ca4634-5648-402b-aaef-13f8d4ee2b4b-agent-certs\") pod \"konnectivity-agent-pd8nn\" (UID: \"72ca4634-5648-402b-aaef-13f8d4ee2b4b\") " pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910487    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-k8s-cni-cncf-io\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910489    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.910535    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret podName:570c7233-f4c2-4c16-b9d9-67baa3fa7b96 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.410525202 +0000 UTC m=+21.173958794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret") pod "global-pull-secret-syncer-vsx66" (UID: "570c7233-f4c2-4c16-b9d9-67baa3fa7b96") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910537    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-registration-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysconfig\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-lib-modules\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdgft\" (UniqueName: \"kubernetes.io/projected/25220e90-f330-4bd6-b2ff-45eb20b344c2-kube-api-access-wdgft\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-netns\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910495    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.910967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-systemd\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-systemd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-tmp-dir\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-dbus\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910740    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-netns\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910750    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-systemd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6b5\" (UniqueName: \"kubernetes.io/projected/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-kube-api-access-6q6b5\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910768    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-systemd\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910705    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-lib-modules\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910708    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysconfig\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910816    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.910968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-ovn\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cnibin\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-var-lib-kubelet\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-tmp\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.911695 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911116    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cnibin\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911140    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-dbus\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911141    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-ovn\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-slash\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911264    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-var-lib-kubelet\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cni-binary-copy\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911341    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-slash\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtmnw\" (UniqueName: \"kubernetes.io/projected/99d52cb6-4109-40d5-8fc7-fb6679a03d60-kube-api-access-mtmnw\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911404    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c640c634-2704-48e7-a121-e0975ae43e23-iptables-alerter-script\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-etc-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911551    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-env-overrides\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-bin\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/528f5273-2e64-4225-b59b-1616674a15fa-ovn-node-metrics-cert\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911654    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-etc-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-socket-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911856    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-bin\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911912    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-socket-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.912436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.912057    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.913269 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.912134    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-script-lib\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.913269 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.912293    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99d52cb6-4109-40d5-8fc7-fb6679a03d60-serviceca\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.913269 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.912326    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-tmp-dir\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:18.913269 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.912603    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cni-binary-copy\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.913269 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.912747    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-daemon-config\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.913269 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.911730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.913560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-device-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.913560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.913560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbffg\" (UniqueName: \"kubernetes.io/projected/cff7b24a-4b98-4592-9344-5ec4930c6079-kube-api-access-jbffg\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:18.913560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-multus-certs\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.913560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72ca4634-5648-402b-aaef-13f8d4ee2b4b-konnectivity-ca\") pod \"konnectivity-agent-pd8nn\" (UID: \"72ca4634-5648-402b-aaef-13f8d4ee2b4b\") " pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:18.913560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-conf\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.913560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d52cb6-4109-40d5-8fc7-fb6679a03d60-host\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.913560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.913560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-log-socket\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.913560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913551    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-log-socket\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.914076 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.913604    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-device-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.914125 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.914111    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.914323 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.914301    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.914386 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.914361    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-multus-certs\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.914435 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.914409    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d52cb6-4109-40d5-8fc7-fb6679a03d60-host\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.914504 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.914477    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-conf\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.914664 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.914645    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-tuned\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.914726 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.914692    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-tmp\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.914834 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.914809    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/528f5273-2e64-4225-b59b-1616674a15fa-ovn-node-metrics-cert\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.914962 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.914928    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72ca4634-5648-402b-aaef-13f8d4ee2b4b-konnectivity-ca\") pod \"konnectivity-agent-pd8nn\" (UID: \"72ca4634-5648-402b-aaef-13f8d4ee2b4b\") " pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:18.915008 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.914993    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72ca4634-5648-402b-aaef-13f8d4ee2b4b-agent-certs\") pod \"konnectivity-agent-pd8nn\" (UID: \"72ca4634-5648-402b-aaef-13f8d4ee2b4b\") " pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:18.938227 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.938207    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmv6\" (UniqueName: \"kubernetes.io/projected/528f5273-2e64-4225-b59b-1616674a15fa-kube-api-access-xrmv6\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:18.941338 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.941313    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6b5\" (UniqueName: \"kubernetes.io/projected/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-kube-api-access-6q6b5\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:18.941656 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.941634    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qcz\" (UniqueName: \"kubernetes.io/projected/c640c634-2704-48e7-a121-e0975ae43e23-kube-api-access-v9qcz\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.941736 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.941634    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9jp\" (UniqueName: \"kubernetes.io/projected/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-kube-api-access-cs9jp\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:18.941736 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.941643    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtmnw\" (UniqueName: \"kubernetes.io/projected/99d52cb6-4109-40d5-8fc7-fb6679a03d60-kube-api-access-mtmnw\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.943923 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.943902    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:43:18.944016 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.943930    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:43:18.944016 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.943958    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7dkm for pod openshift-network-diagnostics/network-check-target-57d72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:18.944135 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:18.944018    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm podName:9df3afdd-f643-4485-a85d-f78211c7b169 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.444001819 +0000 UTC m=+21.207435424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g7dkm" (UniqueName: "kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm") pod "network-check-target-57d72" (UID: "9df3afdd-f643-4485-a85d-f78211c7b169") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:18.946292 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.946273    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbffg\" (UniqueName: \"kubernetes.io/projected/cff7b24a-4b98-4592-9344-5ec4930c6079-kube-api-access-jbffg\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:18.946375 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.946311    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsdb\" (UniqueName: \"kubernetes.io/projected/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-kube-api-access-vdsdb\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:18.949258 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.949243    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdgft\" (UniqueName: \"kubernetes.io/projected/25220e90-f330-4bd6-b2ff-45eb20b344c2-kube-api-access-wdgft\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:18.954603 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.954586    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfr8\" (UniqueName: \"kubernetes.io/projected/094d4ae0-f6f8-44ae-a658-03f6322a3223-kube-api-access-8nfr8\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:18.963452 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.963431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:43:18.971965 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.971949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:43:18.984161 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:18.984145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:43:19.000995 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.000979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:43:19.007531 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.007516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zf8tf"
Nov 02 13:43:19.009847 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:43:19.009828    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc640c634_2704_48e7_a121_e0975ae43e23.slice/crio-57e708d0e6968e5778e6c8d9fc57a25dafc54276a08d45046aceffad61bc9bbc WatchSource:0}: Error finding container 57e708d0e6968e5778e6c8d9fc57a25dafc54276a08d45046aceffad61bc9bbc: Status 404 returned error can't find the container with id 57e708d0e6968e5778e6c8d9fc57a25dafc54276a08d45046aceffad61bc9bbc
Nov 02 13:43:19.012031 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.012014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:19.015601 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.015584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:19.017521 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.017415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:43:19.022905 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.022880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:43:19.028828 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:43:19.028806    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528f5273_2e64_4225_b59b_1616674a15fa.slice/crio-c6b8b6f9d33ef0223dfa8aed556326d8065e19523786af4d8218f089e61ab414 WatchSource:0}: Error finding container c6b8b6f9d33ef0223dfa8aed556326d8065e19523786af4d8218f089e61ab414: Status 404 returned error can't find the container with id c6b8b6f9d33ef0223dfa8aed556326d8065e19523786af4d8218f089e61ab414
Nov 02 13:43:19.029675 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:43:19.029601    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25220e90_f330_4bd6_b2ff_45eb20b344c2.slice/crio-5b42e8d447d6ecc4ed5b3bda6b6c794dcc8677a9e471dec0a385157b6dfb997c WatchSource:0}: Error finding container 5b42e8d447d6ecc4ed5b3bda6b6c794dcc8677a9e471dec0a385157b6dfb997c: Status 404 returned error can't find the container with id 5b42e8d447d6ecc4ed5b3bda6b6c794dcc8677a9e471dec0a385157b6dfb997c
Nov 02 13:43:19.035414 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:43:19.035393    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac48fd07_ce48_4fc0_afa0_e9b31baaaec4.slice/crio-536d7e13a4d6066e9d1b3f035fb781c9209ba1564aa8244a9290c820fb4e8c74 WatchSource:0}: Error finding container 536d7e13a4d6066e9d1b3f035fb781c9209ba1564aa8244a9290c820fb4e8c74: Status 404 returned error can't find the container with id 536d7e13a4d6066e9d1b3f035fb781c9209ba1564aa8244a9290c820fb4e8c74
Nov 02 13:43:19.053208 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.053191    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:19.062989 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.062947    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal" podStartSLOduration=1.062919266 podStartE2EDuration="1.062919266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:19.062575063 +0000 UTC m=+20.826008678" watchObservedRunningTime="2026-11-02 13:43:19.062919266 +0000 UTC m=+20.826352877"
Nov 02 13:43:19.416923 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.416886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:19.417097 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.416961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:19.417097 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:19.417043    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:19.417179 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:19.417107    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret podName:570c7233-f4c2-4c16-b9d9-67baa3fa7b96 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:20.417091851 +0000 UTC m=+22.180525478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret") pod "global-pull-secret-syncer-vsx66" (UID: "570c7233-f4c2-4c16-b9d9-67baa3fa7b96") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:19.417179 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:19.417043    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:19.417179 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:19.417177    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs podName:cff7b24a-4b98-4592-9344-5ec4930c6079 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:20.41716519 +0000 UTC m=+22.180598795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs") pod "network-metrics-daemon-jcqf2" (UID: "cff7b24a-4b98-4592-9344-5ec4930c6079") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:19.518316 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.518250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:19.518492 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:19.518475    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:43:19.518533 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:19.518504    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:43:19.518566 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:19.518536    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7dkm for pod openshift-network-diagnostics/network-check-target-57d72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:19.518612 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:19.518599    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm podName:9df3afdd-f643-4485-a85d-f78211c7b169 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:20.518582932 +0000 UTC m=+22.282016541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7dkm" (UniqueName: "kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm") pod "network-check-target-57d72" (UID: "9df3afdd-f643-4485-a85d-f78211c7b169") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:19.703638 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.703521    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:38:18 +0000 UTC" deadline="2028-03-27 17:58:09.003077287 +0000 UTC"
Nov 02 13:43:19.703638 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.703593    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12268h14m49.299491099s"
Nov 02 13:43:19.891239 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.891127    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgnfh" event={"ID":"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4","Type":"ContainerStarted","Data":"536d7e13a4d6066e9d1b3f035fb781c9209ba1564aa8244a9290c820fb4e8c74"}
Nov 02 13:43:19.894364 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.894291    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xgnrc" event={"ID":"25220e90-f330-4bd6-b2ff-45eb20b344c2","Type":"ContainerStarted","Data":"5b42e8d447d6ecc4ed5b3bda6b6c794dcc8677a9e471dec0a385157b6dfb997c"}
Nov 02 13:43:19.898255 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.898163    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" event={"ID":"528f5273-2e64-4225-b59b-1616674a15fa","Type":"ContainerStarted","Data":"c6b8b6f9d33ef0223dfa8aed556326d8065e19523786af4d8218f089e61ab414"}
Nov 02 13:43:19.904834 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.904797    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5jbwx" event={"ID":"35941a8e-d549-436f-b3e1-0be3ef2cc3a7","Type":"ContainerStarted","Data":"de4fef001fb2f5113dd2fffac2d210041a30b797d5ea7e62bc921410bdc4ac02"}
Nov 02 13:43:19.934962 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.917803    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k" event={"ID":"094d4ae0-f6f8-44ae-a658-03f6322a3223","Type":"ContainerStarted","Data":"050b71e53ba9388f7f10491ddf8541792adb9ee3ee2dda1ca442951deb1670cf"}
Nov 02 13:43:19.934962 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.924595    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4j587" event={"ID":"c640c634-2704-48e7-a121-e0975ae43e23","Type":"ContainerStarted","Data":"57e708d0e6968e5778e6c8d9fc57a25dafc54276a08d45046aceffad61bc9bbc"}
Nov 02 13:43:19.939871 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.939150    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal_d005c7e1a378e0971dc1c77f9a40f556/setup/0.log"
Nov 02 13:43:19.939871 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.939185    2581 generic.go:356] "Generic (PLEG): container finished" podID="d005c7e1a378e0971dc1c77f9a40f556" containerID="065fc2231ae9c9877bb03017332062bf802feae90307084906dbfd48f2bb358d" exitCode=0
Nov 02 13:43:19.939871 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.939245    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal" event={"ID":"d005c7e1a378e0971dc1c77f9a40f556","Type":"ContainerDied","Data":"065fc2231ae9c9877bb03017332062bf802feae90307084906dbfd48f2bb358d"}
Nov 02 13:43:19.939871 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.939297    2581 scope.go:122] "RemoveContainer" containerID="1f239c01336c55275fe87e7b32eeaab468bf85f53b8ce8f8b111b74cb0ed7410"
Nov 02 13:43:19.939871 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.939666    2581 scope.go:122] "RemoveContainer" containerID="1f239c01336c55275fe87e7b32eeaab468bf85f53b8ce8f8b111b74cb0ed7410"
Nov 02 13:43:19.952560 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.952531    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pd8nn" event={"ID":"72ca4634-5648-402b-aaef-13f8d4ee2b4b","Type":"ContainerStarted","Data":"d5475d929c3a5746a5d83f5e066c55dc74590c80a87b3b094717def719f9e3fa"}
Nov 02 13:43:19.955479 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.955400    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zf8tf" event={"ID":"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb","Type":"ContainerStarted","Data":"618b183d511f453e7c50efb4daa7077edd613cd91938fe7db120f64c3fef9f9b"}
Nov 02 13:43:19.965078 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:19.965055    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gkg2v" event={"ID":"99d52cb6-4109-40d5-8fc7-fb6679a03d60","Type":"ContainerStarted","Data":"9fd06223f5e9f1f30d521b3ccae9f526baf46ea53418b559eca50d7dabc9623f"}
Nov 02 13:43:20.011564 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.011225    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal_openshift-machine-config-operator_d005c7e1a378e0971dc1c77f9a40f556_0 in pod sandbox d2b0f73d19c2fa0514d6a23b654b88058c0e04a50f1af46d30f711cbbcc2277f from index: no such id: '1f239c01336c55275fe87e7b32eeaab468bf85f53b8ce8f8b111b74cb0ed7410'" containerID="1f239c01336c55275fe87e7b32eeaab468bf85f53b8ce8f8b111b74cb0ed7410"
Nov 02 13:43:20.011564 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.011286    2581 kuberuntime_container.go:987] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal_openshift-machine-config-operator_d005c7e1a378e0971dc1c77f9a40f556_0 in pod sandbox d2b0f73d19c2fa0514d6a23b654b88058c0e04a50f1af46d30f711cbbcc2277f from index: no such id: '1f239c01336c55275fe87e7b32eeaab468bf85f53b8ce8f8b111b74cb0ed7410'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal_openshift-machine-config-operator(d005c7e1a378e0971dc1c77f9a40f556)\"" logger="UnhandledError"
Nov 02 13:43:20.425065 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:20.425034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:20.425246 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:20.425105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:20.425246 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.425242    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:20.425357 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.425303    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs podName:cff7b24a-4b98-4592-9344-5ec4930c6079 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:22.425278721 +0000 UTC m=+24.188712319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs") pod "network-metrics-daemon-jcqf2" (UID: "cff7b24a-4b98-4592-9344-5ec4930c6079") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:20.425691 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.425671    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:20.425767 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.425727    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret podName:570c7233-f4c2-4c16-b9d9-67baa3fa7b96 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:22.425712889 +0000 UTC m=+24.189146485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret") pod "global-pull-secret-syncer-vsx66" (UID: "570c7233-f4c2-4c16-b9d9-67baa3fa7b96") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:20.525791 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:20.525751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:20.526068 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.526030    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:43:20.526068 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.526076    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:43:20.526195 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.526089    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7dkm for pod openshift-network-diagnostics/network-check-target-57d72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:20.526195 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.526149    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm podName:9df3afdd-f643-4485-a85d-f78211c7b169 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:22.526131394 +0000 UTC m=+24.289564992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7dkm" (UniqueName: "kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm") pod "network-check-target-57d72" (UID: "9df3afdd-f643-4485-a85d-f78211c7b169") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:20.704302 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:20.704258    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:38:18 +0000 UTC" deadline="2028-07-27 12:24:55.576629012 +0000 UTC"
Nov 02 13:43:20.704302 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:20.704301    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15190h41m34.872332909s"
Nov 02 13:43:20.839360 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:20.837061    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:20.839360 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:20.837104    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:20.839360 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.837239    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:20.839360 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:20.837673    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:20.839360 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.837829    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:20.839360 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:20.838427    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:20.987448 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:20.987411    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal" event={"ID":"d005c7e1a378e0971dc1c77f9a40f556","Type":"ContainerStarted","Data":"a802a589d0528862b6df75d668167c7c962add104ad97257e4febe388f262ee3"}
Nov 02 13:43:21.997133 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:21.996319    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="78aa95da86238feda4ef29da8bef3f6a2db2c0b629c58ad50581585b61d97f83" exitCode=0
Nov 02 13:43:21.997133 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:21.996520    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgnfh" event={"ID":"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4","Type":"ContainerDied","Data":"78aa95da86238feda4ef29da8bef3f6a2db2c0b629c58ad50581585b61d97f83"}
Nov 02 13:43:22.059848 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:22.059792    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal" podStartSLOduration=4.059773889 podStartE2EDuration="4.059773889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:21.03367347 +0000 UTC m=+22.797107086" watchObservedRunningTime="2026-11-02 13:43:22.059773889 +0000 UTC m=+23.823207504"
Nov 02 13:43:22.441892 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:22.441612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:22.441892 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:22.441682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:22.441892 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.441807    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:22.441892 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.441869    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret podName:570c7233-f4c2-4c16-b9d9-67baa3fa7b96 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:26.441851277 +0000 UTC m=+28.205284872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret") pod "global-pull-secret-syncer-vsx66" (UID: "570c7233-f4c2-4c16-b9d9-67baa3fa7b96") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:22.442280 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.442003    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:22.442280 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.442044    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs podName:cff7b24a-4b98-4592-9344-5ec4930c6079 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:26.442031911 +0000 UTC m=+28.205465508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs") pod "network-metrics-daemon-jcqf2" (UID: "cff7b24a-4b98-4592-9344-5ec4930c6079") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:22.542138 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:22.542098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:22.542335 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.542310    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:43:22.542335 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.542333    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:43:22.542422 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.542347    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7dkm for pod openshift-network-diagnostics/network-check-target-57d72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:22.542460 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.542419    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm podName:9df3afdd-f643-4485-a85d-f78211c7b169 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:26.542399062 +0000 UTC m=+28.305832677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7dkm" (UniqueName: "kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm") pod "network-check-target-57d72" (UID: "9df3afdd-f643-4485-a85d-f78211c7b169") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:22.837071 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:22.837030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:22.837245 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:22.837143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:22.837245 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:22.837034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:22.837245 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.837227    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:22.837451 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.837141    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:22.837451 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:22.837300    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:24.009857 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:24.009804    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pd8nn" event={"ID":"72ca4634-5648-402b-aaef-13f8d4ee2b4b","Type":"ContainerStarted","Data":"6e682b9264f5f1ca721f67492d74c33a937315dab2a2bcfecb666b8ec1266b27"}
Nov 02 13:43:24.013015 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:24.012990    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:24.013694 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:24.013649    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:24.054003 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:24.053564    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pd8nn" podStartSLOduration=8.215161994 podStartE2EDuration="12.053546019s" totalImagesPullingTime="3.838384025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:24.052775229 +0000 UTC m=+25.816208839" watchObservedRunningTime="2026-11-02 13:43:24.053546019 +0000 UTC m=+25.816979634"
Nov 02 13:43:24.837090 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:24.837029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:24.837223 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:24.837148    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:24.837502 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:24.837482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:24.837595 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:24.837577    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:24.837643 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:24.837633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:24.837742 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:24.837712    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:25.014132 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:25.014095    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k" event={"ID":"094d4ae0-f6f8-44ae-a658-03f6322a3223","Type":"ContainerStarted","Data":"0b16a9a8078105a59ac144e38d08e321376ea4e1655323acfc8db1ca1622ef92"}
Nov 02 13:43:25.019320 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:25.019288    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gkg2v" event={"ID":"99d52cb6-4109-40d5-8fc7-fb6679a03d60","Type":"ContainerStarted","Data":"213be741d9a9d6503632a512721ecf76880aefd66a6e65a11a88c0dd7fd58cfe"}
Nov 02 13:43:26.021067 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.021041    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:43:26.231529 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.231501    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 13:43:26.477694 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.477665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:26.477858 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.477728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:26.477858 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.477850    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:26.478012 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.477917    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs podName:cff7b24a-4b98-4592-9344-5ec4930c6079 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.477898978 +0000 UTC m=+36.241332584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs") pod "network-metrics-daemon-jcqf2" (UID: "cff7b24a-4b98-4592-9344-5ec4930c6079") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:26.478352 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.478299    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:26.478464 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.478355    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret podName:570c7233-f4c2-4c16-b9d9-67baa3fa7b96 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.478340311 +0000 UTC m=+36.241773904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret") pod "global-pull-secret-syncer-vsx66" (UID: "570c7233-f4c2-4c16-b9d9-67baa3fa7b96") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:26.578250 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.578222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:26.578427 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.578402    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:43:26.578427 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.578425    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:43:26.578539 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.578437    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7dkm for pod openshift-network-diagnostics/network-check-target-57d72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:26.578539 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.578481    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm podName:9df3afdd-f643-4485-a85d-f78211c7b169 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.578465669 +0000 UTC m=+36.341899278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7dkm" (UniqueName: "kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm") pod "network-check-target-57d72" (UID: "9df3afdd-f643-4485-a85d-f78211c7b169") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:26.835045 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.835014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:26.835206 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.835142    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:26.835540 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.835521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:26.835657 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.835628    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:26.835778 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.835711    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:26.835851 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:26.835794    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:26.867595 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.867511    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:43:26.231520881Z","UUID":"16415d16-6130-43de-863a-73fc3cea9e12","Handler":null,"Name":"","Endpoint":""}
Nov 02 13:43:26.869508 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.869482    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 13:43:26.869622 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:26.869520    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 13:43:27.030522 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:27.030154    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5jbwx" event={"ID":"35941a8e-d549-436f-b3e1-0be3ef2cc3a7","Type":"ContainerStarted","Data":"11d8899868339fa05f6b57b155797fb1d956b861fac00ae72f94fab45e015942"}
Nov 02 13:43:27.032850 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:27.032823    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k" event={"ID":"094d4ae0-f6f8-44ae-a658-03f6322a3223","Type":"ContainerStarted","Data":"af43bc13e61064887f09d2003af2fb8a4af1f846b067a8f4b19ffc4717809214"}
Nov 02 13:43:27.034666 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:27.034618    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zf8tf" event={"ID":"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb","Type":"ContainerStarted","Data":"7e52ed2a1f38a70182b350e51a716f39bdad6041931e58cc541a8540f9ba299d"}
Nov 02 13:43:27.037660 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:27.037634    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="b277846e1624087cff0bd5ffc9481ab561f7046ef784ea952b202d935235ed6e" exitCode=0
Nov 02 13:43:27.037756 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:27.037671    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgnfh" event={"ID":"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4","Type":"ContainerDied","Data":"b277846e1624087cff0bd5ffc9481ab561f7046ef784ea952b202d935235ed6e"}
Nov 02 13:43:27.078839 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:27.078221    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5jbwx" podStartSLOduration=7.7142331219999996 podStartE2EDuration="15.078204432s" totalImagesPullingTime="7.36397131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:27.077721999 +0000 UTC m=+28.841155632" watchObservedRunningTime="2026-11-02 13:43:27.078204432 +0000 UTC m=+28.841638047"
Nov 02 13:43:27.078839 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:27.078703    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gkg2v" podStartSLOduration=10.116340405 podStartE2EDuration="15.078691157s" totalImagesPullingTime="4.962350752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:25.065444514 +0000 UTC m=+26.828878130" watchObservedRunningTime="2026-11-02 13:43:27.078691157 +0000 UTC m=+28.842124770"
Nov 02 13:43:27.176123 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:27.175855    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zf8tf" podStartSLOduration=7.495167214 podStartE2EDuration="15.175842606s" totalImagesPullingTime="7.680675392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:27.124239571 +0000 UTC m=+28.887673185" watchObservedRunningTime="2026-11-02 13:43:27.175842606 +0000 UTC m=+28.939276219"
Nov 02 13:43:28.042771 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:28.042615    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="4460650740d3db7c959ae72e778f2250ace9db959063e479e2beab394c0f28b0" exitCode=0
Nov 02 13:43:28.042771 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:28.042658    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgnfh" event={"ID":"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4","Type":"ContainerDied","Data":"4460650740d3db7c959ae72e778f2250ace9db959063e479e2beab394c0f28b0"}
Nov 02 13:43:28.046703 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:28.046676    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k" event={"ID":"094d4ae0-f6f8-44ae-a658-03f6322a3223","Type":"ContainerStarted","Data":"355a65d154edbc4ca0c3fe2cbea3b30d3d5befc370c860b3a22d00a23a8ffe1b"}
Nov 02 13:43:28.118872 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:28.118843    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:28.119043 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:28.118978    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:43:28.119716 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:28.119694    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:43:28.198716 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:28.198674    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k" podStartSLOduration=8.342653497 podStartE2EDuration="16.198657231s" totalImagesPullingTime="7.856003734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:28.153348335 +0000 UTC m=+29.916781950" watchObservedRunningTime="2026-11-02 13:43:28.198657231 +0000 UTC m=+29.962090845"
Nov 02 13:43:28.836153 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:28.836126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:28.836153 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:28.836143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:28.836375 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:28.836245    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:28.836424 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:28.836382    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:28.836482 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:28.836460    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:28.836560 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:28.836543    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:29.051164 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:29.051127    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="62d32104f3ace1ee342a972c733e744cc2435763720ee35ac4e0f5e80d6a76fa" exitCode=0
Nov 02 13:43:29.051626 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:29.051211    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgnfh" event={"ID":"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4","Type":"ContainerDied","Data":"62d32104f3ace1ee342a972c733e744cc2435763720ee35ac4e0f5e80d6a76fa"}
Nov 02 13:43:29.053270 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:29.053167    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xgnrc" event={"ID":"25220e90-f330-4bd6-b2ff-45eb20b344c2","Type":"ContainerStarted","Data":"aea10cde2be531868e9c26125ee14dd5430c0404feb516b854ea10fa60adea29"}
Nov 02 13:43:29.054868 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:29.054846    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4j587" event={"ID":"c640c634-2704-48e7-a121-e0975ae43e23","Type":"ContainerStarted","Data":"9b686d7e3d1904b06132ee0fb85716657ad8cc6c835b09f4c9da9bbac3566759"}
Nov 02 13:43:29.152121 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:29.152071    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xgnrc" podStartSLOduration=7.504002114 podStartE2EDuration="17.152055708s" totalImagesPullingTime="9.648053594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:29.151906886 +0000 UTC m=+30.915340501" watchObservedRunningTime="2026-11-02 13:43:29.152055708 +0000 UTC m=+30.915489322"
Nov 02 13:43:30.558833 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:30.558813    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5jbwx_35941a8e-d549-436f-b3e1-0be3ef2cc3a7/dns-node-resolver/0.log"
Nov 02 13:43:30.834923 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:30.834891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:30.835110 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:30.834946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:30.835110 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:30.834969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:30.835110 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:30.835099    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:30.835274 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:30.835154    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:30.835274 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:30.835257    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:31.062186 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:31.062149    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="07a6872ca12066354044b3a01b3b3b6bbfd4ab481c341c7f16b65b611d54ef15" exitCode=0
Nov 02 13:43:31.062399 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:31.062194    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgnfh" event={"ID":"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4","Type":"ContainerDied","Data":"07a6872ca12066354044b3a01b3b3b6bbfd4ab481c341c7f16b65b611d54ef15"}
Nov 02 13:43:31.149436 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:31.149391    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4j587" podStartSLOduration=11.187889187 podStartE2EDuration="19.149371033s" totalImagesPullingTime="7.961481846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:29.197551891 +0000 UTC m=+30.960985504" watchObservedRunningTime="2026-11-02 13:43:31.149371033 +0000 UTC m=+32.912804647"
Nov 02 13:43:31.875992 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:31.875962    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gkg2v_99d52cb6-4109-40d5-8fc7-fb6679a03d60/node-ca/0.log"
Nov 02 13:43:32.067803 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:32.067771    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="d0091d56433015751224a7ce390ce21f36f1af62bd41de07017f6091b5a600db" exitCode=0
Nov 02 13:43:32.068005 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:32.067830    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgnfh" event={"ID":"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4","Type":"ContainerDied","Data":"d0091d56433015751224a7ce390ce21f36f1af62bd41de07017f6091b5a600db"}
Nov 02 13:43:32.834950 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:32.834911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:32.835124 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:32.834956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:32.835124 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:32.835010    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:32.835124 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:32.835087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:32.835295 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:32.835209    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:32.835350 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:32.835314    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:33.075212 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:33.075082    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cgnfh" event={"ID":"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4","Type":"ContainerStarted","Data":"14162a44d33bf31c2dc4167e218cb836c43659adc68656fb3173ed9c711a2e77"}
Nov 02 13:43:33.077150 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:33.077120    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" event={"ID":"528f5273-2e64-4225-b59b-1616674a15fa","Type":"ContainerStarted","Data":"c78ed3cbeaeb6f130b668079f603cf70ab1362ffab495e8f3d0606ce4b1a2985"}
Nov 02 13:43:33.077237 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:33.077192    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" event={"ID":"528f5273-2e64-4225-b59b-1616674a15fa","Type":"ContainerStarted","Data":"e884dc0ba1c3ede486209561e9860f4a3e0998982f74953c1e11f20cdc9bf065"}
Nov 02 13:43:33.077300 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:33.077243    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" event={"ID":"528f5273-2e64-4225-b59b-1616674a15fa","Type":"ContainerStarted","Data":"78dc409d685866b83278f79ccef301827d53813c4c85cc3e23b54cd2e97fc47d"}
Nov 02 13:43:33.126673 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:33.126634    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cgnfh" podStartSLOduration=11.518702159 podStartE2EDuration="21.126621041s" totalImagesPullingTime="9.607918882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:33.125600567 +0000 UTC m=+34.889034183" watchObservedRunningTime="2026-11-02 13:43:33.126621041 +0000 UTC m=+34.890054655"
Nov 02 13:43:34.082124 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:34.082084    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" event={"ID":"528f5273-2e64-4225-b59b-1616674a15fa","Type":"ContainerStarted","Data":"1cd019abcbcd4d59e096fe5e9f6f7d7925d09124ba7f53d3c1b5a4c5aae87dc7"}
Nov 02 13:43:34.082617 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:34.082205    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" event={"ID":"528f5273-2e64-4225-b59b-1616674a15fa","Type":"ContainerStarted","Data":"9136d5975f1a8ed175602a135d46db087bc58e0c9741c57d7a9deaec7e39add1"}
Nov 02 13:43:34.082617 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:34.082275    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" event={"ID":"528f5273-2e64-4225-b59b-1616674a15fa","Type":"ContainerStarted","Data":"e90c60fa04d3a9a5648ad2983733c2edd752c011545caaafae43e5586c3535bc"}
Nov 02 13:43:34.537516 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:34.537483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:34.537680 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:34.537536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:34.537680 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.537621    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:34.537680 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.537623    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:34.537680 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.537674    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs podName:cff7b24a-4b98-4592-9344-5ec4930c6079 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:50.537656761 +0000 UTC m=+52.301090357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs") pod "network-metrics-daemon-jcqf2" (UID: "cff7b24a-4b98-4592-9344-5ec4930c6079") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:34.537817 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.537688    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret podName:570c7233-f4c2-4c16-b9d9-67baa3fa7b96 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:50.537681956 +0000 UTC m=+52.301115549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret") pod "global-pull-secret-syncer-vsx66" (UID: "570c7233-f4c2-4c16-b9d9-67baa3fa7b96") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:34.638293 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:34.638262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:34.638448 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.638373    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:43:34.638448 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.638388    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:43:34.638448 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.638396    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7dkm for pod openshift-network-diagnostics/network-check-target-57d72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:34.638448 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.638437    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm podName:9df3afdd-f643-4485-a85d-f78211c7b169 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:50.638426005 +0000 UTC m=+52.401859610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7dkm" (UniqueName: "kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm") pod "network-check-target-57d72" (UID: "9df3afdd-f643-4485-a85d-f78211c7b169") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:34.834786 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:34.834711    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:34.834786 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:34.834758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:34.835009 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.834877    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:34.835009 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.834927    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:34.835009 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:34.834974    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:34.835149 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:34.835069    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:36.089562 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:36.089520    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" event={"ID":"528f5273-2e64-4225-b59b-1616674a15fa","Type":"ContainerStarted","Data":"1ffdc329f6c18c24e1b9b15e226f969cab438df6947ef6de5637416f42567968"}
Nov 02 13:43:36.838054 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:36.838023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:36.838238 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:36.838129    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:36.838238 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:36.838154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:36.838238 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:36.838147    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:36.838238 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:36.838234    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:36.838440 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:36.838306    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:38.096501 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:38.096462    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" event={"ID":"528f5273-2e64-4225-b59b-1616674a15fa","Type":"ContainerStarted","Data":"95ef30495f8ff8a5bb48b7c83e12c6de554060c84ffe8175ec456d25d217df99"}
Nov 02 13:43:38.097055 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:38.096718    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:38.112088 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:38.112063    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:38.205556 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:38.205504    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" podStartSLOduration=12.356957202 podStartE2EDuration="26.205491174s" totalImagesPullingTime="13.848533972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:38.205338235 +0000 UTC m=+39.968771860" watchObservedRunningTime="2026-11-02 13:43:38.205491174 +0000 UTC m=+39.968924787"
Nov 02 13:43:38.835762 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:38.835728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:38.835976 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:38.835895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:38.835976 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:38.835919    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:38.836075 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:38.835971    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:38.836075 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:38.836002    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:38.836075 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:38.836067    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:39.099394 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:39.099313    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:43:39.099789 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:39.099763    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:39.115828 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:39.115797    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:40.103182 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:40.103146    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:43:40.835324 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:40.835283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:40.835324 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:40.835307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:40.835584 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:40.835395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:40.835584 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:40.835415    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:40.835584 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:40.835470    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:40.835584 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:40.835543    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:41.105394 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:41.105316    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:43:42.835300 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:42.835260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:42.835917 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:42.835269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:42.835917 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:42.835402    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:42.835917 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:42.835419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:42.835917 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:42.835531    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:42.835917 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:42.835629    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:44.838108 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:44.838073    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:44.838638 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:44.838173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:44.838638 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:44.838201    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:44.838638 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:44.838248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:44.838638 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:44.838328    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:44.838638 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:44.838401    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:45.965008 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:45.964962    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:43:45.965452 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:45.965175    2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:43:45.980083 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:45.980030    2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" podUID="528f5273-2e64-4225-b59b-1616674a15fa" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 02 13:43:45.989447 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:45.989405    2581 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw" podUID="528f5273-2e64-4225-b59b-1616674a15fa" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 02 13:43:46.835690 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:46.835647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:46.835690 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:46.835678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:46.835996 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:46.835710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:46.835996 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:46.835793    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:46.835996 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:46.835837    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:46.835996 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:46.835909    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:48.836352 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:48.836306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:48.836909 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:48.836395    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:48.836909 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:48.836472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:48.836909 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:48.836568    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:48.836909 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:48.836649    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:48.836909 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:48.836649    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:50.545186 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:50.545090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:50.545584 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:50.545186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:50.545584 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.545249    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:50.545584 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.545292    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:50.545584 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.545336    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret podName:570c7233-f4c2-4c16-b9d9-67baa3fa7b96 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:22.545312022 +0000 UTC m=+84.308745636 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret") pod "global-pull-secret-syncer-vsx66" (UID: "570c7233-f4c2-4c16-b9d9-67baa3fa7b96") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:50.545584 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.545354    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs podName:cff7b24a-4b98-4592-9344-5ec4930c6079 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:22.545346084 +0000 UTC m=+84.308779679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs") pod "network-metrics-daemon-jcqf2" (UID: "cff7b24a-4b98-4592-9344-5ec4930c6079") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:50.615440 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:50.615343    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vsx66"]
Nov 02 13:43:50.616041 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:50.615532    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:50.616041 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.615656    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:50.616852 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:50.616825    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jcqf2"]
Nov 02 13:43:50.616998 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:50.616981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:50.617191 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.617102    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:50.617608 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:50.617586    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-57d72"]
Nov 02 13:43:50.617712 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:50.617699    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:50.617815 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.617796    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:50.646303 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:50.646259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:50.646461 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.646389    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:43:50.646461 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.646403    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:43:50.646461 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.646412    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7dkm for pod openshift-network-diagnostics/network-check-target-57d72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:50.646461 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:50.646461    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm podName:9df3afdd-f643-4485-a85d-f78211c7b169 nodeName:}" failed. No retries permitted until 2026-11-02 13:44:22.646448777 +0000 UTC m=+84.409882383 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7dkm" (UniqueName: "kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm") pod "network-check-target-57d72" (UID: "9df3afdd-f643-4485-a85d-f78211c7b169") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:51.835022 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:51.834754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:51.835500 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:51.834754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:51.835500 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:51.835076    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:51.835500 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:51.835114    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:52.217341 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:52.217266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5jbwx_35941a8e-d549-436f-b3e1-0be3ef2cc3a7/dns-node-resolver/0.log"
Nov 02 13:43:52.834798 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:52.834756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:52.835027 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:52.834886    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:53.817156 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:53.817127    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gkg2v_99d52cb6-4109-40d5-8fc7-fb6679a03d60/node-ca/0.log"
Nov 02 13:43:53.834579 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:53.834536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:53.834579 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:53.834562    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:53.834826 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:53.834647    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-57d72" podUID="9df3afdd-f643-4485-a85d-f78211c7b169"
Nov 02 13:43:53.834826 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:53.834714    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vsx66" podUID="570c7233-f4c2-4c16-b9d9-67baa3fa7b96"
Nov 02 13:43:54.838193 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:54.838164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:54.838581 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:43:54.838279    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jcqf2" podUID="cff7b24a-4b98-4592-9344-5ec4930c6079"
Nov 02 13:43:55.435430 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.435341    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeReady"
Nov 02 13:43:55.435604 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.435491    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 13:43:55.524330 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.524295    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n2d6j"]
Nov 02 13:43:55.526153 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.526133    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lq2nc"]
Nov 02 13:43:55.526272 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.526252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:43:55.528055 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.528033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.556440 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.556410    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lq2nc"]
Nov 02 13:43:55.557204 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.557181    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n2d6j"]
Nov 02 13:43:55.558775 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.558747    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 13:43:55.558955 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.558798    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 13:43:55.558955 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.558798    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 13:43:55.558955 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.558915    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 13:43:55.559154 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.559138    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hkz22\""
Nov 02 13:43:55.559154 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.559144    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 13:43:55.559448 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.559422    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6wff6\""
Nov 02 13:43:55.560424 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.560404    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lrzvt"]
Nov 02 13:43:55.562318 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.562301    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.580845 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.580810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5rq\" (UniqueName: \"kubernetes.io/projected/8e76a365-0c53-4369-b49e-b19a6611a391-kube-api-access-bx5rq\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.581028 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.580862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e76a365-0c53-4369-b49e-b19a6611a391-metrics-tls\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.581028 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.580930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkg7\" (UniqueName: \"kubernetes.io/projected/55837b60-e43f-4eb2-9505-c755bde739ec-kube-api-access-fjkg7\") pod \"ingress-canary-n2d6j\" (UID: \"55837b60-e43f-4eb2-9505-c755bde739ec\") " pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:43:55.581028 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.580997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e76a365-0c53-4369-b49e-b19a6611a391-config-volume\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.581028 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.581014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e76a365-0c53-4369-b49e-b19a6611a391-tmp-dir\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.581163 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.581095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55837b60-e43f-4eb2-9505-c755bde739ec-cert\") pod \"ingress-canary-n2d6j\" (UID: \"55837b60-e43f-4eb2-9505-c755bde739ec\") " pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:43:55.584238 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.584209    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lrzvt"]
Nov 02 13:43:55.655401 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.655371    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 13:43:55.655401 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.655372    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 13:43:55.655623 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.655427    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 13:43:55.655623 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.655471    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9sg29\""
Nov 02 13:43:55.655623 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.655504    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 13:43:55.681389 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e6c927e4-b288-490a-acb9-c8e61846da1e-crio-socket\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.681389 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c927e4-b288-490a-acb9-c8e61846da1e-data-volume\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.681616 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e76a365-0c53-4369-b49e-b19a6611a391-metrics-tls\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.681616 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681514    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkg7\" (UniqueName: \"kubernetes.io/projected/55837b60-e43f-4eb2-9505-c755bde739ec-kube-api-access-fjkg7\") pod \"ingress-canary-n2d6j\" (UID: \"55837b60-e43f-4eb2-9505-c755bde739ec\") " pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:43:55.681616 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5x76\" (UniqueName: \"kubernetes.io/projected/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-api-access-c5x76\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.681616 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e76a365-0c53-4369-b49e-b19a6611a391-config-volume\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.681616 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e76a365-0c53-4369-b49e-b19a6611a391-tmp-dir\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.681616 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55837b60-e43f-4eb2-9505-c755bde739ec-cert\") pod \"ingress-canary-n2d6j\" (UID: \"55837b60-e43f-4eb2-9505-c755bde739ec\") " pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:43:55.681842 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.681842 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6c927e4-b288-490a-acb9-c8e61846da1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.681842 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.681728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5rq\" (UniqueName: \"kubernetes.io/projected/8e76a365-0c53-4369-b49e-b19a6611a391-kube-api-access-bx5rq\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.682351 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.682327    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e76a365-0c53-4369-b49e-b19a6611a391-tmp-dir\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.682666 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.682640    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e76a365-0c53-4369-b49e-b19a6611a391-config-volume\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.687138 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.687071    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55837b60-e43f-4eb2-9505-c755bde739ec-cert\") pod \"ingress-canary-n2d6j\" (UID: \"55837b60-e43f-4eb2-9505-c755bde739ec\") " pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:43:55.687138 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.687114    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e76a365-0c53-4369-b49e-b19a6611a391-metrics-tls\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.707861 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.707821    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkg7\" (UniqueName: \"kubernetes.io/projected/55837b60-e43f-4eb2-9505-c755bde739ec-kube-api-access-fjkg7\") pod \"ingress-canary-n2d6j\" (UID: \"55837b60-e43f-4eb2-9505-c755bde739ec\") " pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:43:55.707861 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.707840    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5rq\" (UniqueName: \"kubernetes.io/projected/8e76a365-0c53-4369-b49e-b19a6611a391-kube-api-access-bx5rq\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.783092 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.783030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e6c927e4-b288-490a-acb9-c8e61846da1e-crio-socket\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.783092 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.783094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c927e4-b288-490a-acb9-c8e61846da1e-data-volume\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.783311 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.783137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5x76\" (UniqueName: \"kubernetes.io/projected/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-api-access-c5x76\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.783311 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.783178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.783311 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.783209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6c927e4-b288-490a-acb9-c8e61846da1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.783463 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.783309    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e6c927e4-b288-490a-acb9-c8e61846da1e-crio-socket\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.783554 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.783533    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c927e4-b288-490a-acb9-c8e61846da1e-data-volume\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.783818 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.783794    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.785480 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.785463    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6c927e4-b288-490a-acb9-c8e61846da1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.811394 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.811355    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5x76\" (UniqueName: \"kubernetes.io/projected/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-api-access-c5x76\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:55.834954 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.834901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:43:55.835148 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.835136    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:43:55.837784 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.837760    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:43:55.844710 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.844685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:55.852302 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.852242    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 13:43:55.852433 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.852419    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fs4v4\""
Nov 02 13:43:55.852512 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.852492    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xwmgj\""
Nov 02 13:43:55.852787 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.852769    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 13:43:55.852866 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.852851    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 13:43:55.875019 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:55.874981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:43:56.163610 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:56.163583    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lq2nc"]
Nov 02 13:43:56.166753 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:43:56.166722    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e76a365_0c53_4369_b49e_b19a6611a391.slice/crio-894c6eb722903e8f2cb1d6f76d661a42268a0af22297a52d690c7d123178f1ec WatchSource:0}: Error finding container 894c6eb722903e8f2cb1d6f76d661a42268a0af22297a52d690c7d123178f1ec: Status 404 returned error can't find the container with id 894c6eb722903e8f2cb1d6f76d661a42268a0af22297a52d690c7d123178f1ec
Nov 02 13:43:56.175098 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:56.175070    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n2d6j"]
Nov 02 13:43:56.177992 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:43:56.177972    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55837b60_e43f_4eb2_9505_c755bde739ec.slice/crio-e3180528833f71f0fcf8ffcd0c204bb3c10ebfc60236bc48c4387db0de0cb46e WatchSource:0}: Error finding container e3180528833f71f0fcf8ffcd0c204bb3c10ebfc60236bc48c4387db0de0cb46e: Status 404 returned error can't find the container with id e3180528833f71f0fcf8ffcd0c204bb3c10ebfc60236bc48c4387db0de0cb46e
Nov 02 13:43:56.203029 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:56.202929    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lrzvt"]
Nov 02 13:43:56.206466 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:43:56.206279    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c927e4_b288_490a_acb9_c8e61846da1e.slice/crio-922aaf5edd9aa62864ffa50b6185f0e8166a495caa6979645c07bb5cc06414d3 WatchSource:0}: Error finding container 922aaf5edd9aa62864ffa50b6185f0e8166a495caa6979645c07bb5cc06414d3: Status 404 returned error can't find the container with id 922aaf5edd9aa62864ffa50b6185f0e8166a495caa6979645c07bb5cc06414d3
Nov 02 13:43:56.840153 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:56.839542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:43:56.856404 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:56.856367    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lj29d\""
Nov 02 13:43:56.857018 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:56.856417    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 13:43:57.143967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:57.143853    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrzvt" event={"ID":"e6c927e4-b288-490a-acb9-c8e61846da1e","Type":"ContainerStarted","Data":"205e7906bdeee20dd9295fd9ce8b020a5100ee032db3947c53812ad6d90f4f68"}
Nov 02 13:43:57.143967 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:57.143956    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrzvt" event={"ID":"e6c927e4-b288-490a-acb9-c8e61846da1e","Type":"ContainerStarted","Data":"25b256a900ee83e86c1de04193e5d76643f636c70ec600f8fe042d8378bea8ad"}
Nov 02 13:43:57.144456 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:57.143992    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrzvt" event={"ID":"e6c927e4-b288-490a-acb9-c8e61846da1e","Type":"ContainerStarted","Data":"922aaf5edd9aa62864ffa50b6185f0e8166a495caa6979645c07bb5cc06414d3"}
Nov 02 13:43:57.145299 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:57.145272    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lq2nc" event={"ID":"8e76a365-0c53-4369-b49e-b19a6611a391","Type":"ContainerStarted","Data":"894c6eb722903e8f2cb1d6f76d661a42268a0af22297a52d690c7d123178f1ec"}
Nov 02 13:43:57.146777 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:57.146750    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n2d6j" event={"ID":"55837b60-e43f-4eb2-9505-c755bde739ec","Type":"ContainerStarted","Data":"e3180528833f71f0fcf8ffcd0c204bb3c10ebfc60236bc48c4387db0de0cb46e"}
Nov 02 13:43:57.971777 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:57.971741    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"]
Nov 02 13:43:57.975693 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:57.974981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.015377 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.015340    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 13:43:58.015377 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.015358    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 13:43:58.015613 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.015431    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 13:43:58.015613 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.015445    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 13:43:58.015613 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.015450    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 13:43:58.015613 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.015488    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f4zv8\""
Nov 02 13:43:58.016439 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.016415    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"]
Nov 02 13:43:58.052031 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.051994    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"]
Nov 02 13:43:58.055037 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.054304    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jgpx4"]
Nov 02 13:43:58.055037 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.054431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.058193 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.056796    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.075718 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.075579    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"]
Nov 02 13:43:58.084104 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.084077    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 13:43:58.084609 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.084589    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 13:43:58.084858 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.084815    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4d48r\""
Nov 02 13:43:58.085229 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.085209    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 13:43:58.085358 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.085343    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hnbmg\""
Nov 02 13:43:58.085487 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.085473    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 13:43:58.101688 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.101636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srg7z\" (UniqueName: \"kubernetes.io/projected/380def42-3dee-46c1-83b8-52910cc74811-kube-api-access-srg7z\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.101927 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.101695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/380def42-3dee-46c1-83b8-52910cc74811-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.101927 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.101763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.101927 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.101865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.102348 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.102321    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 13:43:58.102627 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.102603    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 13:43:58.151520 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.151480    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lq2nc" event={"ID":"8e76a365-0c53-4369-b49e-b19a6611a391","Type":"ContainerStarted","Data":"dafddd568e0c1b51a78afd897d75cf2396b82ee37c3592d46741b117d5b9b878"}
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-wtmp\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srg7z\" (UniqueName: \"kubernetes.io/projected/380def42-3dee-46c1-83b8-52910cc74811-kube-api-access-srg7z\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-root\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-textfile\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-sys\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b0175cfb-9276-42ed-bff8-f359a5727eb7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/380def42-3dee-46c1-83b8-52910cc74811-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.220023 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-tls\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.221366 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-accelerators-collector-config\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.221366 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cczzh\" (UniqueName: \"kubernetes.io/projected/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-kube-api-access-cczzh\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.221366 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.205894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-metrics-client-ca\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.221366 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.210920    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/380def42-3dee-46c1-83b8-52910cc74811-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.221366 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.211040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56w8p\" (UniqueName: \"kubernetes.io/projected/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-api-access-56w8p\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.221366 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.211496    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.221366 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.211653    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.255307 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.255270    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srg7z\" (UniqueName: \"kubernetes.io/projected/380def42-3dee-46c1-83b8-52910cc74811-kube-api-access-srg7z\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.292063 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.292017    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:43:58.312252 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.312252 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.312470 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b0175cfb-9276-42ed-bff8-f359a5727eb7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.312470 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.312470 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-tls\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.312470 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-accelerators-collector-config\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.312470 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cczzh\" (UniqueName: \"kubernetes.io/projected/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-kube-api-access-cczzh\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.312470 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-metrics-client-ca\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.312470 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56w8p\" (UniqueName: \"kubernetes.io/projected/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-api-access-56w8p\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.312470 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-wtmp\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.312745 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.312745 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-root\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.312745 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-textfile\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.312745 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-sys\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.312745 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.312603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.314515 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.314025    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.314515 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.314156    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-metrics-client-ca\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.314685 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.314627    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-wtmp\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.315408 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.315189    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b0175cfb-9276-42ed-bff8-f359a5727eb7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.315408 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.315324    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.315408 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.315328    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-accelerators-collector-config\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.315408 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.315364    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-root\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.315408 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.315408    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-sys\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.315705 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.315451    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-textfile\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.316112 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.316068    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.317091 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.316671    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.317091 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.317051    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-tls\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.318622 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.318583    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.339373 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.339304    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56w8p\" (UniqueName: \"kubernetes.io/projected/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-api-access-56w8p\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.339538 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.339403    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cczzh\" (UniqueName: \"kubernetes.io/projected/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-kube-api-access-cczzh\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.394099 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.392893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:43:58.398590 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.398566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:43:58.414370 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:43:58.414319    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad8eb6b_fa80_41f6_b5ff_51c39bba446c.slice/crio-4fade1d8ef298325be12da5e63e05078f7fa8f4c9d45e3649e4b6bdd86fc8374 WatchSource:0}: Error finding container 4fade1d8ef298325be12da5e63e05078f7fa8f4c9d45e3649e4b6bdd86fc8374: Status 404 returned error can't find the container with id 4fade1d8ef298325be12da5e63e05078f7fa8f4c9d45e3649e4b6bdd86fc8374
Nov 02 13:43:58.640766 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.640727    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"]
Nov 02 13:43:58.712482 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:58.712445    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"]
Nov 02 13:43:58.716405 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:43:58.716371    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0175cfb_9276_42ed_bff8_f359a5727eb7.slice/crio-32a102b0508db4ce09b08788497e2387b3e6b14f2cd8dd0a0dd5b950eff44736 WatchSource:0}: Error finding container 32a102b0508db4ce09b08788497e2387b3e6b14f2cd8dd0a0dd5b950eff44736: Status 404 returned error can't find the container with id 32a102b0508db4ce09b08788497e2387b3e6b14f2cd8dd0a0dd5b950eff44736
Nov 02 13:43:59.159576 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.159363    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb" event={"ID":"380def42-3dee-46c1-83b8-52910cc74811","Type":"ContainerStarted","Data":"9a7a702f6a6f6579582a1f4023301151b373241b49311946977dccadd48d2b4f"}
Nov 02 13:43:59.160528 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.159616    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb" event={"ID":"380def42-3dee-46c1-83b8-52910cc74811","Type":"ContainerStarted","Data":"94088038578e4095c3d098c86d9dfcf156756936ad3f07f56d81acac0f1be53d"}
Nov 02 13:43:59.160528 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.159647    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb" event={"ID":"380def42-3dee-46c1-83b8-52910cc74811","Type":"ContainerStarted","Data":"5954afd4612bdd401080fe703e06ba03b4c27cd8f5bee438c63058155fe78e2d"}
Nov 02 13:43:59.163332 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.163262    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lrzvt" event={"ID":"e6c927e4-b288-490a-acb9-c8e61846da1e","Type":"ContainerStarted","Data":"d8c774dd86ce1e6c5bd81680c1be20229be2dd75a3783f6fdfc11d729502e334"}
Nov 02 13:43:59.165379 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.165344    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jgpx4" event={"ID":"fad8eb6b-fa80-41f6-b5ff-51c39bba446c","Type":"ContainerStarted","Data":"4fade1d8ef298325be12da5e63e05078f7fa8f4c9d45e3649e4b6bdd86fc8374"}
Nov 02 13:43:59.167209 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.167180    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c" event={"ID":"b0175cfb-9276-42ed-bff8-f359a5727eb7","Type":"ContainerStarted","Data":"32a102b0508db4ce09b08788497e2387b3e6b14f2cd8dd0a0dd5b950eff44736"}
Nov 02 13:43:59.169269 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.169240    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lq2nc" event={"ID":"8e76a365-0c53-4369-b49e-b19a6611a391","Type":"ContainerStarted","Data":"70c82637ac71d973fb8f57a119c486e9b8f8e02639df712119a5dd79a6f041d5"}
Nov 02 13:43:59.169427 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.169409    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:43:59.459885 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.459820    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-lq2nc" podStartSLOduration=2.679298238 podStartE2EDuration="4.459803082s" totalImagesPullingTime="1.780504844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:59.39737646 +0000 UTC m=+61.160810075" watchObservedRunningTime="2026-11-02 13:43:59.459803082 +0000 UTC m=+61.223236696"
Nov 02 13:43:59.460109 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.459922    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lrzvt" podStartSLOduration=2.242033069 podStartE2EDuration="4.459916219s" totalImagesPullingTime="2.21788315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:59.459257709 +0000 UTC m=+61.222691323" watchObservedRunningTime="2026-11-02 13:43:59.459916219 +0000 UTC m=+61.223349833"
Nov 02 13:43:59.499786 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.499748    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:43:59.503605 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.503572    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.521749 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.521715    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 13:43:59.521962 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.521823    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 13:43:59.521962 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.521918    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 13:43:59.522232 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.522213    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 13:43:59.522316 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.522280    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 13:43:59.522430 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.522366    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 13:43:59.522430 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.522417    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 13:43:59.522536 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.522496    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rxjnr\""
Nov 02 13:43:59.522587 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.522578    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 13:43:59.539467 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.539062    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:43:59.539467 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.539427    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 13:43:59.623431 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-out\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623632 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623632 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623632 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623632 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623843 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623843 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623843 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623843 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623843 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrckt\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-kube-api-access-vrckt\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623843 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.623843 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.624239 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.623856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-web-config\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725046 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.724926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725046 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725274 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725274 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725274 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725274 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725274 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725274 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrckt\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-kube-api-access-vrckt\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725274 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725565 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725565 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-web-config\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725565 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-out\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725565 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725378    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.725565 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.725410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.726089 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.726061    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.727216 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.726907    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.732186 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.732099    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.732186 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.732138    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.732186 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.732102    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-web-config\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.732696 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.732638    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.732873 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.732850    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.732966 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.732894    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.732966 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.732916    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.734118 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.734094    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.734266 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.734243    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-out\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.751782 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.751713    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrckt\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-kube-api-access-vrckt\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:43:59.841042 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:43:59.841001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:00.174342 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:00.174147    2581 generic.go:356] "Generic (PLEG): container finished" podID="fad8eb6b-fa80-41f6-b5ff-51c39bba446c" containerID="5bf20c211a48dd042532c014bca001f27375681082c55d0dafb505be4d8173ac" exitCode=0
Nov 02 13:44:00.175181 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:00.174223    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jgpx4" event={"ID":"fad8eb6b-fa80-41f6-b5ff-51c39bba446c","Type":"ContainerDied","Data":"5bf20c211a48dd042532c014bca001f27375681082c55d0dafb505be4d8173ac"}
Nov 02 13:44:00.177714 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:00.177674    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c" event={"ID":"b0175cfb-9276-42ed-bff8-f359a5727eb7","Type":"ContainerStarted","Data":"77eda6856b0d350e7cacea08809cd93bbd1541e2d6ef61772123008a7c42812a"}
Nov 02 13:44:00.179978 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:00.179667    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n2d6j" event={"ID":"55837b60-e43f-4eb2-9505-c755bde739ec","Type":"ContainerStarted","Data":"9cd1694b2aea952a40c6d2822d8e3150549a922a0ea362577c8789729817f0af"}
Nov 02 13:44:00.185747 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:00.185702    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb" event={"ID":"380def42-3dee-46c1-83b8-52910cc74811","Type":"ContainerStarted","Data":"5bc52687deb0488f82870be98113277ba2572f57e62dbb50178f4ecc430b0e97"}
Nov 02 13:44:00.212007 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:00.211979    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:00.227786 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:44:00.227755    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947938ad_88e5_4ebd_aefe_94ca1b32e7f7.slice/crio-9b2ed9769382472c01d45016c359d28f9c73ca0b93ec11ff8681d526316a90f2 WatchSource:0}: Error finding container 9b2ed9769382472c01d45016c359d28f9c73ca0b93ec11ff8681d526316a90f2: Status 404 returned error can't find the container with id 9b2ed9769382472c01d45016c359d28f9c73ca0b93ec11ff8681d526316a90f2
Nov 02 13:44:00.335971 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:00.335518    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n2d6j" podStartSLOduration=2.175684625 podStartE2EDuration="5.33549221s" totalImagesPullingTime="3.159807585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:00.28497332 +0000 UTC m=+62.048406936" watchObservedRunningTime="2026-11-02 13:44:00.33549221 +0000 UTC m=+62.098925820"
Nov 02 13:44:00.337509 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:00.337156    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb" podStartSLOduration=2.218523271 podStartE2EDuration="3.337134076s" totalImagesPullingTime="1.118610805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:00.333271824 +0000 UTC m=+62.096705440" watchObservedRunningTime="2026-11-02 13:44:00.337134076 +0000 UTC m=+62.100567692"
Nov 02 13:44:01.192386 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:01.192111    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jgpx4" event={"ID":"fad8eb6b-fa80-41f6-b5ff-51c39bba446c","Type":"ContainerStarted","Data":"85952e2408b397206051167af8483879cd496733c5720868c69eaa6990bac4bd"}
Nov 02 13:44:01.192816 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:01.192411    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jgpx4" event={"ID":"fad8eb6b-fa80-41f6-b5ff-51c39bba446c","Type":"ContainerStarted","Data":"5e5e5648002a3a9f015bc35c433d9930243bfecb92e88b7de44d658715b91d7b"}
Nov 02 13:44:01.194653 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:01.194623    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c" event={"ID":"b0175cfb-9276-42ed-bff8-f359a5727eb7","Type":"ContainerStarted","Data":"46fecdd24caab427fe636021ab2d61a66e1f8214325a0629a78024054b3ae606"}
Nov 02 13:44:01.194815 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:01.194685    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c" event={"ID":"b0175cfb-9276-42ed-bff8-f359a5727eb7","Type":"ContainerStarted","Data":"ec9537c792b3ffc279e88c387d58ac70fc53d71d90c1198ccd6edab0e5c92185"}
Nov 02 13:44:01.196008 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:01.195981    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerStarted","Data":"9b2ed9769382472c01d45016c359d28f9c73ca0b93ec11ff8681d526316a90f2"}
Nov 02 13:44:01.242337 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:01.242286    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jgpx4" podStartSLOduration=2.161267498 podStartE2EDuration="3.242271313s" totalImagesPullingTime="986.695407ms" totalInitContainerRuntime="94.308408ms" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:01.241345203 +0000 UTC m=+63.004778841" watchObservedRunningTime="2026-11-02 13:44:01.242271313 +0000 UTC m=+63.005705007"
Nov 02 13:44:01.292651 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:01.292596    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c" podStartSLOduration=1.900016451 podStartE2EDuration="3.292577473s" totalImagesPullingTime="1.392561022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:01.291583112 +0000 UTC m=+63.055016727" watchObservedRunningTime="2026-11-02 13:44:01.292577473 +0000 UTC m=+63.056011087"
Nov 02 13:44:02.201856 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.201811    2581 generic.go:356] "Generic (PLEG): container finished" podID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerID="e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741" exitCode=0
Nov 02 13:44:02.202307 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.201904    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerDied","Data":"e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741"}
Nov 02 13:44:02.766816 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.766769    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-566f4c7487-mbcfx"]
Nov 02 13:44:02.774889 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.774856    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.797260 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.797225    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-566f4c7487-mbcfx"]
Nov 02 13:44:02.798398 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.798354    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5tjtq\""
Nov 02 13:44:02.798539 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.798461    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 13:44:02.798539 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.798463    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 13:44:02.809337 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.809312    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 13:44:02.809337 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.809312    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 13:44:02.809588 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.809573    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-97ql1c6l88l52\""
Nov 02 13:44:02.860313 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.860271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-tls\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.860313 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.860322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-client-ca-bundle\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.860586 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.860351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7dc2b96a-b979-4de1-a69b-ce1c566518c6-audit-log\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.860586 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.860451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.860586 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.860564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-metrics-server-audit-profiles\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.860704 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.860644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-client-certs\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.860743 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.860716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9222\" (UniqueName: \"kubernetes.io/projected/7dc2b96a-b979-4de1-a69b-ce1c566518c6-kube-api-access-q9222\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.962299 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.962249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9222\" (UniqueName: \"kubernetes.io/projected/7dc2b96a-b979-4de1-a69b-ce1c566518c6-kube-api-access-q9222\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.962509 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.962335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-tls\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.962509 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.962364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-client-ca-bundle\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.962509 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.962404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7dc2b96a-b979-4de1-a69b-ce1c566518c6-audit-log\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.962509 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.962433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.962509 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.962485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-metrics-server-audit-profiles\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.962768 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.962539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-client-certs\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.963068 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.963016    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7dc2b96a-b979-4de1-a69b-ce1c566518c6-audit-log\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.963559 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.963483    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.963723 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.963702    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-metrics-server-audit-profiles\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.966005 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.965979    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-client-certs\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.966148 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.966127    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-tls\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.966207 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.966136    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-client-ca-bundle\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:02.991647 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:02.991608    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9222\" (UniqueName: \"kubernetes.io/projected/7dc2b96a-b979-4de1-a69b-ce1c566518c6-kube-api-access-q9222\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:03.088818 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:03.088716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:03.417364 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:03.417340    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-566f4c7487-mbcfx"]
Nov 02 13:44:03.419597 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:44:03.419553    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc2b96a_b979_4de1_a69b_ce1c566518c6.slice/crio-eee4840a3e24b7d7fa57f555724ac0ee49a296dc595ca316657ce3d202d72c3f WatchSource:0}: Error finding container eee4840a3e24b7d7fa57f555724ac0ee49a296dc595ca316657ce3d202d72c3f: Status 404 returned error can't find the container with id eee4840a3e24b7d7fa57f555724ac0ee49a296dc595ca316657ce3d202d72c3f
Nov 02 13:44:04.213979 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:04.213906    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx" event={"ID":"7dc2b96a-b979-4de1-a69b-ce1c566518c6","Type":"ContainerStarted","Data":"eee4840a3e24b7d7fa57f555724ac0ee49a296dc595ca316657ce3d202d72c3f"}
Nov 02 13:44:04.219288 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:04.219251    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerStarted","Data":"b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e"}
Nov 02 13:44:04.219441 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:04.219343    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerStarted","Data":"a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4"}
Nov 02 13:44:04.219441 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:04.219386    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerStarted","Data":"d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b"}
Nov 02 13:44:04.219441 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:04.219425    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerStarted","Data":"6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19"}
Nov 02 13:44:04.219562 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:04.219451    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerStarted","Data":"0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4"}
Nov 02 13:44:05.027874 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.027839    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 13:44:05.037113 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.037062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.072218 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.072184    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 13:44:05.072421 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.072374    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 13:44:05.072487 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.072429    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 13:44:05.073709 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.073691    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 13:44:05.100224 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.100184    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 13:44:05.100480 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.100460    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 13:44:05.126390 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126346    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 13:44:05.126596 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126570    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 13:44:05.126686 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126598    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 13:44:05.126686 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126599    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7our0gqq5qckd\""
Nov 02 13:44:05.126766 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126693    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 13:44:05.126766 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126724    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 13:44:05.126822 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126787    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 13:44:05.126850 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126834    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 13:44:05.126850 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126844    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 13:44:05.126908 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126885    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 13:44:05.126908 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.126899    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xv47k\""
Nov 02 13:44:05.181131 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181131 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-config-out\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181352 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181352 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181455 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181530 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181565 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-config\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181595 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181595 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181677 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181751 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181813 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-web-config\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181870 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.181997 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.181976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.182059 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.182017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjndc\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-kube-api-access-bjndc\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.182059 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.182040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.182128 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.182058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.182128 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.182072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.182340 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.182323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.182408 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.182389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.226274 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.226228    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerStarted","Data":"5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83"}
Nov 02 13:44:05.227906 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.227875    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx" event={"ID":"7dc2b96a-b979-4de1-a69b-ce1c566518c6","Type":"ContainerStarted","Data":"00c7853d14e188229da9fb406860ae00e8df015c38a2f108faa91807cbe5757f"}
Nov 02 13:44:05.283194 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283194 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283194 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283459 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283459 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283459 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283608 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-config-out\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283608 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283608 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283748 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283748 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283748 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-config\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.283748 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284034 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284034 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284034 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284034 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283841    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284034 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-web-config\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284034 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284034 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.283975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284034 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.284007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjndc\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-kube-api-access-bjndc\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284571 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.284095    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284571 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.284121    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284571 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.284399    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.284983 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.284910    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.286779 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.285404    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.286779 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.286720    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.287855 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.287658    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.288250 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.288223    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.288516 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.288496    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.289387 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.289360    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.290010 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.289504    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.290010 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.289586    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.290010 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.289652    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.290010 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.289809    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-config-out\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.290010 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.289826    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-config\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.290010 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.289884    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.290010 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.290004    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-web-config\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.290840 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.290819    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.321951 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.321916    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjndc\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-kube-api-access-bjndc\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.347945 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.347890    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.435896129 podStartE2EDuration="6.347873815s" totalImagesPullingTime="2.898014031s" totalInitContainerRuntime="13.963655ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:05.295322639 +0000 UTC m=+67.058756253" watchObservedRunningTime="2026-11-02 13:44:05.347873815 +0000 UTC m=+67.111307428"
Nov 02 13:44:05.348124 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.348071    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx" podStartSLOduration=2.101004492 podStartE2EDuration="3.348064492s" totalImagesPullingTime="1.24706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:05.346772237 +0000 UTC m=+67.110205850" watchObservedRunningTime="2026-11-02 13:44:05.348064492 +0000 UTC m=+67.111498106"
Nov 02 13:44:05.355792 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.355752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:05.698283 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:05.698198    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 13:44:05.701385 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:44:05.701353    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5bed06_f76d_474a_88ed_1fc6be553533.slice/crio-293dddafbc0ac9b18beefb2d7c6b1ae612e0a25d3f68aaa89a0f5671c5ca083b WatchSource:0}: Error finding container 293dddafbc0ac9b18beefb2d7c6b1ae612e0a25d3f68aaa89a0f5671c5ca083b: Status 404 returned error can't find the container with id 293dddafbc0ac9b18beefb2d7c6b1ae612e0a25d3f68aaa89a0f5671c5ca083b
Nov 02 13:44:06.232762 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:06.232723    2581 generic.go:356] "Generic (PLEG): container finished" podID="da5bed06-f76d-474a-88ed-1fc6be553533" containerID="e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504" exitCode=0
Nov 02 13:44:06.233175 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:06.232808    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerDied","Data":"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"}
Nov 02 13:44:06.233175 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:06.232898    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerStarted","Data":"293dddafbc0ac9b18beefb2d7c6b1ae612e0a25d3f68aaa89a0f5671c5ca083b"}
Nov 02 13:44:09.189581 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:09.189545    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:44:09.248964 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:09.248902    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerStarted","Data":"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"}
Nov 02 13:44:09.249240 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:09.249037    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerStarted","Data":"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"}
Nov 02 13:44:11.271660 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:11.271625    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerStarted","Data":"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"}
Nov 02 13:44:11.342754 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:11.340768    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pd8nn_72ca4634-5648-402b-aaef-13f8d4ee2b4b/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:11.609720 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:11.609690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pd8nn_72ca4634-5648-402b-aaef-13f8d4ee2b4b/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:12.266989 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:12.266957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pd8nn_72ca4634-5648-402b-aaef-13f8d4ee2b4b/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:12.278195 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:12.278158    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerStarted","Data":"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"}
Nov 02 13:44:12.278593 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:12.278227    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerStarted","Data":"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"}
Nov 02 13:44:12.278593 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:12.278257    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerStarted","Data":"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"}
Nov 02 13:44:12.352578 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:12.352522    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.543325647 podStartE2EDuration="8.352507253s" totalImagesPullingTime="4.779562577s" totalInitContainerRuntime="29.619029ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:12.351587823 +0000 UTC m=+74.115021436" watchObservedRunningTime="2026-11-02 13:44:12.352507253 +0000 UTC m=+74.115940870"
Nov 02 13:44:15.356484 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:15.356443    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:15.991350 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:15.991317    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:22.561312 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.561171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:44:22.561312 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.561234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:22.576854 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.576822    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 13:44:22.584225 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.584191    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:44:22.590355 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.590326    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 13:44:22.594565 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.594538    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:22.662081 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.662037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:44:22.670984 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.670960    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lj29d\""
Nov 02 13:44:22.676147 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.676118    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:44:22.677471 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.677450    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 13:44:22.697811 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.697782    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 13:44:22.705435 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.705403    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:44:22.862647 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.862563    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fs4v4\""
Nov 02 13:44:22.867622 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.867589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:22.868469 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.868446    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xwmgj\""
Nov 02 13:44:22.874050 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.874013    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:44:22.983610 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:22.983486    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jcqf2"]
Nov 02 13:44:22.987161 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:44:22.987133    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff7b24a_4b98_4592_9344_5ec4930c6079.slice/crio-fed22b7ba9098af776862ae593258d6a4982a7d8192b859fa9f92354edd721ae WatchSource:0}: Error finding container fed22b7ba9098af776862ae593258d6a4982a7d8192b859fa9f92354edd721ae: Status 404 returned error can't find the container with id fed22b7ba9098af776862ae593258d6a4982a7d8192b859fa9f92354edd721ae
Nov 02 13:44:23.089567 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:23.089527    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:23.089718 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:23.089604    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:23.169063 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:23.168889    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vsx66"]
Nov 02 13:44:23.171718 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:44:23.171691    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570c7233_f4c2_4c16_b9d9_67baa3fa7b96.slice/crio-bba513efe642ce0839e826b411cb03d094ba2fd4c1308e0758e196d6abfad6e7 WatchSource:0}: Error finding container bba513efe642ce0839e826b411cb03d094ba2fd4c1308e0758e196d6abfad6e7: Status 404 returned error can't find the container with id bba513efe642ce0839e826b411cb03d094ba2fd4c1308e0758e196d6abfad6e7
Nov 02 13:44:23.174511 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:23.174440    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-57d72"]
Nov 02 13:44:23.178102 ip-10-0-131-98 kubenswrapper[2581]: W1102 13:44:23.178074    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df3afdd_f643_4485_a85d_f78211c7b169.slice/crio-11783e1bcf6f17ce82b0adedfa657e45f78a89daebd7e34771e77c1bef5659ee WatchSource:0}: Error finding container 11783e1bcf6f17ce82b0adedfa657e45f78a89daebd7e34771e77c1bef5659ee: Status 404 returned error can't find the container with id 11783e1bcf6f17ce82b0adedfa657e45f78a89daebd7e34771e77c1bef5659ee
Nov 02 13:44:23.317047 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:23.317013    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-57d72" event={"ID":"9df3afdd-f643-4485-a85d-f78211c7b169","Type":"ContainerStarted","Data":"11783e1bcf6f17ce82b0adedfa657e45f78a89daebd7e34771e77c1bef5659ee"}
Nov 02 13:44:23.318282 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:23.318253    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vsx66" event={"ID":"570c7233-f4c2-4c16-b9d9-67baa3fa7b96","Type":"ContainerStarted","Data":"bba513efe642ce0839e826b411cb03d094ba2fd4c1308e0758e196d6abfad6e7"}
Nov 02 13:44:23.319395 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:23.319372    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jcqf2" event={"ID":"cff7b24a-4b98-4592-9344-5ec4930c6079","Type":"ContainerStarted","Data":"fed22b7ba9098af776862ae593258d6a4982a7d8192b859fa9f92354edd721ae"}
Nov 02 13:44:24.325963 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:24.325902    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jcqf2" event={"ID":"cff7b24a-4b98-4592-9344-5ec4930c6079","Type":"ContainerStarted","Data":"071456c24ae6bb71825d28f49db14f67ef2ff4aef51606146a170e2c34bff749"}
Nov 02 13:44:24.326569 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:24.326008    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jcqf2" event={"ID":"cff7b24a-4b98-4592-9344-5ec4930c6079","Type":"ContainerStarted","Data":"15198236d73cec4408d6c52ef8e9ccac7a8a8bce4046e59f636a4e029ae6a97c"}
Nov 02 13:44:24.370699 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:24.370629    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jcqf2" podStartSLOduration=71.451629938 podStartE2EDuration="1m12.370612789s" totalImagesPullingTime="918.982851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:24.369870374 +0000 UTC m=+86.133303986" watchObservedRunningTime="2026-11-02 13:44:24.370612789 +0000 UTC m=+86.134046403"
Nov 02 13:44:25.913168 ip-10-0-131-98 kubenswrapper[2581]: E1102 13:44:25.913083    2581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:44:27.340864 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:27.340824    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-57d72" event={"ID":"9df3afdd-f643-4485-a85d-f78211c7b169","Type":"ContainerStarted","Data":"1b404f01fb696224e99ad3fd837a6658aec477ffb37e181f3558b86b79d6448d"}
Nov 02 13:44:27.341341 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:27.340911    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:44:27.385129 ip-10-0-131-98 kubenswrapper[2581]: I1102 13:44:27.385061    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-57d72" podStartSLOduration=72.156891936 podStartE2EDuration="1m15.385044349s" totalImagesPullingTime="3.228152413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:27.384296367 +0000 UTC m=+89.147729982" watchObservedRunningTime="2026-11-02 13:44:27.385044349 +0000 UTC m=+89.148478024"
Nov 02 13:44:27.979290 ip-10-0-131-98 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 13:44:27.991431 ip-10-0-131-98 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 13:44:27.991616 ip-10-0-131-98 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 13:44:27.991811 ip-10-0-131-98 systemd[1]: kubelet.service: Consumed 4.776s CPU time.
Nov 02 13:44:28.005670 ip-10-0-131-98 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 13:44:28.074198 ip-10-0-131-98 kubenswrapper[6284]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:44:28.074198 ip-10-0-131-98 kubenswrapper[6284]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 13:44:28.074198 ip-10-0-131-98 kubenswrapper[6284]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:44:28.074198 ip-10-0-131-98 kubenswrapper[6284]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:44:28.076467 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076365    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:44:28.076467 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076466    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:44:28.076467 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076471    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:44:28.076467 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076474    6284 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076477    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076480    6284 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076483    6284 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076486    6284 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076488    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076491    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076494    6284 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076496    6284 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076499    6284 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076503    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076506    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076517    6284 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076520    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076523    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076526    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076528    6284 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076531    6284 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076533    6284 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076536    6284 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:44:28.076596 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076539    6284 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076541    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076544    6284 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076546    6284 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076549    6284 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076552    6284 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076554    6284 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076557    6284 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076560    6284 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076563    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076566    6284 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076569    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076572    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076577    6284 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076610    6284 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076615    6284 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076618    6284 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076621    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076625    6284 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:44:28.077109 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076628    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076631    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076635    6284 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076639    6284 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076642    6284 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076646    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076649    6284 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076651    6284 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076654    6284 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076657    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076659    6284 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076662    6284 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076664    6284 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076667    6284 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076670    6284 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076673    6284 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076676    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076678    6284 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076682    6284 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:44:28.077566 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076685    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076688    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076690    6284 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076693    6284 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076695    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076698    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076701    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076704    6284 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076707    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076709    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076712    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076714    6284 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076717    6284 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076720    6284 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076722    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076725    6284 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076727    6284 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076729    6284 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076733    6284 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:44:28.078069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076735    6284 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076738    6284 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076740    6284 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076744    6284 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076747    6284 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076749    6284 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076752    6284 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076755    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076757    6284 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076761    6284 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076764    6284 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076766    6284 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076769    6284 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076771    6284 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076774    6284 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076777    6284 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076779    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076781    6284 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076784    6284 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076787    6284 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:44:28.078542 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076790    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076793    6284 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076796    6284 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076798    6284 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076801    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.076804    6284 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077649    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077657    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077660    6284 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077663    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077666    6284 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077668    6284 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077674    6284 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077677    6284 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077682    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077685    6284 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077688    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077690    6284 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077693    6284 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:44:28.079081 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077696    6284 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077698    6284 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077701    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077704    6284 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077707    6284 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077710    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077712    6284 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077715    6284 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077718    6284 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077720    6284 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077723    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077726    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077728    6284 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077731    6284 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077734    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077736    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077739    6284 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077742    6284 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077744    6284 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:44:28.079546 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077747    6284 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077750    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077753    6284 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077755    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077759    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077762    6284 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077764    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077775    6284 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077778    6284 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077781    6284 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077783    6284 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077786    6284 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077789    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077791    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077794    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077796    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077799    6284 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077802    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077804    6284 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:44:28.080042 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077807    6284 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077810    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077813    6284 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077817    6284 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077822    6284 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077825    6284 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077828    6284 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077830    6284 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077833    6284 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077836    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077839    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077842    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077844    6284 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077847    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077849    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077852    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077864    6284 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077868    6284 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077871    6284 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:44:28.080515 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077874    6284 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077877    6284 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077880    6284 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077883    6284 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077886    6284 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077889    6284 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077892    6284 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077895    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077898    6284 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077901    6284 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077904    6284 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077906    6284 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077909    6284 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077912    6284 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077914    6284 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077917    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077920    6284 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077923    6284 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077926    6284 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:44:28.081018 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077929    6284 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077945    6284 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077947    6284 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077950    6284 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077953    6284 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077956    6284 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077959    6284 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077961    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077964    6284 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077967    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077971    6284 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077974    6284 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077976    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077979    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077981    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077984    6284 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.077987    6284 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078527    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078533    6284 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078536    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:44:28.081494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078540    6284 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078543    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078546    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078549    6284 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078552    6284 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078555    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078557    6284 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078560    6284 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078563    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078565    6284 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078568    6284 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078570    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078573    6284 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078576    6284 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078578    6284 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078581    6284 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078583    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078586    6284 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078588    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078591    6284 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:44:28.081997 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078593    6284 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078596    6284 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078599    6284 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078602    6284 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078604    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078607    6284 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078610    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078612    6284 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078615    6284 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078622    6284 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078625    6284 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078628    6284 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078631    6284 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078633    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078636    6284 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078638    6284 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078641    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078644    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078646    6284 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078648    6284 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:44:28.082491 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078651    6284 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078653    6284 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078656    6284 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078658    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078661    6284 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078664    6284 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078666    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078669    6284 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078672    6284 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078674    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078677    6284 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078679    6284 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078682    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078684    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078687    6284 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078690    6284 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078692    6284 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078695    6284 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078697    6284 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:44:28.082996 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078700    6284 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078704    6284 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078709    6284 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078713    6284 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078716    6284 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078718    6284 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078721    6284 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078723    6284 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078726    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078729    6284 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078732    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078735    6284 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078737    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078740    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078742    6284 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078745    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078747    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078750    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078752    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:44:28.083571 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078755    6284 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078758    6284 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078760    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078763    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078765    6284 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078768    6284 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078770    6284 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078773    6284 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078775    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078778    6284 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078781    6284 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078783    6284 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078786    6284 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078788    6284 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078791    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078795    6284 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078800    6284 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078804    6284 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078807    6284 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078810    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:44:28.084312 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078813    6284 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078816    6284 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078820    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078822    6284 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.078825    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.078959    6284 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.078977    6284 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.078986    6284 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.078991    6284 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.078996    6284 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079000    6284 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079005    6284 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079014    6284 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079017    6284 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079020    6284 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079024    6284 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079028    6284 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079031    6284 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079034    6284 flags.go:64] FLAG: --cgroup-root=""
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079037    6284 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079045    6284 flags.go:64] FLAG: --client-ca-file=""
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079048    6284 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079051    6284 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 13:44:28.084858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079057    6284 flags.go:64] FLAG: --cluster-domain=""
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079060    6284 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079064    6284 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079067    6284 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079072    6284 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079077    6284 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079081    6284 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079085    6284 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079089    6284 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079092    6284 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079096    6284 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079099    6284 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079102    6284 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079105    6284 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079110    6284 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079114    6284 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079117    6284 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079120    6284 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079123    6284 flags.go:64] FLAG: --enable-server="true"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079126    6284 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079132    6284 flags.go:64] FLAG: --event-burst="100"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079135    6284 flags.go:64] FLAG: --event-qps="50"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079139    6284 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079142    6284 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 13:44:28.085483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079146    6284 flags.go:64] FLAG: --eviction-hard=""
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079151    6284 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079154    6284 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079157    6284 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079161    6284 flags.go:64] FLAG: --eviction-soft=""
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079165    6284 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079169    6284 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079172    6284 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079175    6284 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079178    6284 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079181    6284 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079184    6284 flags.go:64] FLAG: --feature-gates=""
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079191    6284 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079194    6284 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079198    6284 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079202    6284 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079205    6284 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079208    6284 flags.go:64] FLAG: --help="false"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079211    6284 flags.go:64] FLAG: --hostname-override="ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079214    6284 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079217    6284 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079220    6284 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079224    6284 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 13:44:28.086166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079228    6284 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079231    6284 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079234    6284 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079237    6284 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079240    6284 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079244    6284 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079247    6284 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079250    6284 flags.go:64] FLAG: --kube-reserved=""
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079253    6284 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079257    6284 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079260    6284 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079262    6284 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079265    6284 flags.go:64] FLAG: --lock-file=""
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079268    6284 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079273    6284 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079276    6284 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079283    6284 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079286    6284 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079289    6284 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079292    6284 flags.go:64] FLAG: --logging-format="text"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079295    6284 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079300    6284 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079304    6284 flags.go:64] FLAG: --manifest-url=""
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079307    6284 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079312    6284 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 13:44:28.086754 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079316    6284 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079321    6284 flags.go:64] FLAG: --max-pods="110"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079324    6284 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079327    6284 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079330    6284 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079333    6284 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079337    6284 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079340    6284 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079343    6284 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079351    6284 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079355    6284 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079358    6284 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079361    6284 flags.go:64] FLAG: --pod-cidr=""
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079364    6284 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079367    6284 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079370    6284 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079373    6284 flags.go:64] FLAG: --port="10250"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079377    6284 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079380    6284 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c719521f980d5365"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079384    6284 flags.go:64] FLAG: --qos-reserved=""
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079387    6284 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079392    6284 flags.go:64] FLAG: --register-node="true"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079395    6284 flags.go:64] FLAG: --register-with-taints=""
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079399    6284 flags.go:64] FLAG: --registry-burst="10"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079402    6284 flags.go:64] FLAG: --registry-qps="5"
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079405    6284 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 13:44:28.087371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079408    6284 flags.go:64] FLAG: --reserved-memory=""
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079412    6284 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079417    6284 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079420    6284 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079423    6284 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079427    6284 flags.go:64] FLAG: --runonce="false"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079430    6284 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079434    6284 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079437    6284 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079440    6284 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079444    6284 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079447    6284 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079450    6284 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079454    6284 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079457    6284 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079461    6284 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079464    6284 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079467    6284 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079470    6284 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079473    6284 flags.go:64] FLAG: --system-cgroups=""
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079476    6284 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079482    6284 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079485    6284 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079488    6284 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079492    6284 flags.go:64] FLAG: --tls-min-version=""
Nov 02 13:44:28.088026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079495    6284 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079498    6284 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079503    6284 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079506    6284 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079509    6284 flags.go:64] FLAG: --v="2"
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079514    6284 flags.go:64] FLAG: --version="false"
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079519    6284 flags.go:64] FLAG: --vmodule=""
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079523    6284 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.079527    6284 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079675    6284 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079679    6284 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079682    6284 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079685    6284 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079688    6284 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079690    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079693    6284 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079696    6284 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079699    6284 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079702    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079705    6284 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079707    6284 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079710    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:44:28.088622 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079713    6284 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079715    6284 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079718    6284 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079721    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079723    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079726    6284 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079728    6284 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079731    6284 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079734    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079737    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079740    6284 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079742    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079747    6284 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079750    6284 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079752    6284 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079755    6284 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079758    6284 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079760    6284 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079763    6284 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:44:28.089222 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079766    6284 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079769    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079772    6284 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079775    6284 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079777    6284 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079780    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079782    6284 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079785    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079788    6284 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079790    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079793    6284 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079796    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079799    6284 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079801    6284 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079803    6284 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079806    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079809    6284 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079812    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079814    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079817    6284 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:44:28.089737 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079819    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079822    6284 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079824    6284 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079827    6284 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079830    6284 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079834    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079837    6284 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079839    6284 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079842    6284 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079845    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079847    6284 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079850    6284 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079853    6284 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079855    6284 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079858    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079861    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079864    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079866    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079869    6284 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:44:28.090293 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079872    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079875    6284 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079877    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079880    6284 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079882    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079885    6284 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079887    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079890    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079892    6284 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079895    6284 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079898    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079900    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079903    6284 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079906    6284 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079910    6284 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079914    6284 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079917    6284 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079920    6284 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079924    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:44:28.090758 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079926    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079929    6284 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079947    6284 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079952    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079956    6284 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079960    6284 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079965    6284 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079969    6284 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079974    6284 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079979    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079983    6284 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079985    6284 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079990    6284 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079994    6284 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.079997    6284 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080001    6284 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:44:28.091240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.080033    6284 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080106    6284 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080113    6284 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080117    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080120    6284 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080123    6284 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080126    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080128    6284 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080131    6284 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080135    6284 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080138    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080141    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080143    6284 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080146    6284 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080149    6284 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080151    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080156    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080159    6284 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080161    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:44:28.091736 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080164    6284 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080167    6284 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080170    6284 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080172    6284 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080175    6284 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080178    6284 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080180    6284 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080183    6284 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080186    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080188    6284 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080191    6284 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080194    6284 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080196    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080200    6284 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080205    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080207    6284 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080210    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080213    6284 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080215    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080218    6284 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:44:28.092277 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080221    6284 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080223    6284 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080226    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080228    6284 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080231    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080234    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080236    6284 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080239    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080241    6284 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080245    6284 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080248    6284 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080251    6284 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080253    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080256    6284 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080259    6284 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080261    6284 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080264    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080267    6284 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080270    6284 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080273    6284 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:44:28.092869 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080275    6284 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080278    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080281    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080283    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080286    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080289    6284 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080293    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080296    6284 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080299    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080301    6284 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080304    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080306    6284 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080309    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080311    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080314    6284 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080317    6284 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080319    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080322    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080324    6284 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:44:28.093537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080328    6284 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080332    6284 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080337    6284 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080340    6284 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080343    6284 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080346    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080349    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080351    6284 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080354    6284 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080356    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080359    6284 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080361    6284 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080364    6284 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080366    6284 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080369    6284 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080371    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080374    6284 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080377    6284 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080381    6284 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:44:28.094044 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080388    6284 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080392    6284 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080396    6284 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080399    6284 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080402    6284 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080404    6284 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080407    6284 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080410    6284 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080413    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.080415    6284 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.085807    6284 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.085830    6284 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.085852    6284 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.085872    6284 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085957    6284 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085966    6284 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085970    6284 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085972    6284 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085975    6284 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085978    6284 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:44:28.094647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085981    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085984    6284 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085986    6284 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085989    6284 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085992    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085996    6284 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.085998    6284 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086001    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086004    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086006    6284 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086009    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086012    6284 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086015    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086018    6284 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086020    6284 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086023    6284 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086025    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086028    6284 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086031    6284 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:44:28.095239 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086034    6284 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086040    6284 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086044    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086046    6284 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086050    6284 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086052    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086055    6284 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086058    6284 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086060    6284 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086063    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086066    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086069    6284 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086071    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086074    6284 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086077    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086079    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086082    6284 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086084    6284 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086087    6284 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:44:28.095896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086090    6284 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086093    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086096    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086099    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086102    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086104    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086107    6284 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086110    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086112    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086115    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086117    6284 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086120    6284 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086123    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086125    6284 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086128    6284 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086130    6284 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086133    6284 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086136    6284 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086138    6284 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086141    6284 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:44:28.096748 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086144    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086146    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086149    6284 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086151    6284 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086154    6284 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086157    6284 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086159    6284 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086162    6284 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086166    6284 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086170    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086174    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086177    6284 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086180    6284 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086183    6284 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086186    6284 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086190    6284 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086194    6284 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086198    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086201    6284 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:44:28.097862 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086204    6284 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086206    6284 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086209    6284 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086212    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086214    6284 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086217    6284 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086220    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086222    6284 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086225    6284 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086228    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086230    6284 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086233    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086235    6284 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086238    6284 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086240    6284 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086243    6284 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086246    6284 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086249    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086252    6284 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086255    6284 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:44:28.098721 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086257    6284 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086260    6284 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086263    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.086295    6284 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086448    6284 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086455    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086458    6284 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086461    6284 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086465    6284 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086469    6284 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086472    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086475    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086477    6284 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086480    6284 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086483    6284 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086486    6284 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:44:28.099715 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086488    6284 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086491    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086494    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086497    6284 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086500    6284 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086503    6284 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086506    6284 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086508    6284 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086511    6284 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086514    6284 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086516    6284 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086519    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086522    6284 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086525    6284 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086530    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086534    6284 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086537    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086540    6284 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086543    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:44:28.100372 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086546    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086550    6284 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086552    6284 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086555    6284 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086557    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086560    6284 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086562    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086566    6284 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086569    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086572    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086575    6284 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086578    6284 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086580    6284 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086583    6284 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086586    6284 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086588    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086591    6284 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086595    6284 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086599    6284 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:44:28.101248 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086602    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086605    6284 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086608    6284 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086611    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086613    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086616    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086619    6284 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086621    6284 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086624    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086626    6284 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086629    6284 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086631    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086634    6284 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086636    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086639    6284 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086641    6284 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086644    6284 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086647    6284 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086649    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:44:28.102162 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086652    6284 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086655    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086658    6284 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086660    6284 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086663    6284 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086665    6284 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086668    6284 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086671    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086673    6284 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086676    6284 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086678    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086681    6284 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086684    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086686    6284 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086689    6284 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086691    6284 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086694    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086696    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086699    6284 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086701    6284 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:44:28.102986 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086704    6284 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086706    6284 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086709    6284 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086712    6284 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086715    6284 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086717    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086720    6284 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086722    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086726    6284 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086728    6284 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086734    6284 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086736    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086739    6284 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086742    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086745    6284 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086753    6284 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:44:28.103537 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086756    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.086780    6284 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086815    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086819    6284 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086823    6284 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086828    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086832    6284 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086835    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086838    6284 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086841    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086845    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086848    6284 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086850    6284 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086853    6284 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086856    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:44:28.103921 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086858    6284 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086861    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086864    6284 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086867    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086869    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086872    6284 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086875    6284 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086877    6284 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086880    6284 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086883    6284 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086885    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086888    6284 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086894    6284 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086898    6284 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086901    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086904    6284 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086907    6284 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086910    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086913    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:44:28.104307 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086916    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086919    6284 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086922    6284 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086924    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086927    6284 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086929    6284 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086945    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086948    6284 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086951    6284 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086953    6284 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086956    6284 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086959    6284 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086962    6284 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086965    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086968    6284 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086970    6284 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086973    6284 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086976    6284 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086979    6284 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086981    6284 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:44:28.104775 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086984    6284 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086986    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086989    6284 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086991    6284 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086994    6284 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.086999    6284 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087002    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087004    6284 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087007    6284 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087010    6284 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087013    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087016    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087019    6284 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087021    6284 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087024    6284 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087027    6284 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087029    6284 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087032    6284 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087036    6284 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087039    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:44:28.105321 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087041    6284 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087044    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087047    6284 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087050    6284 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087052    6284 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087055    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087057    6284 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087060    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087063    6284 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087065    6284 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087068    6284 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087070    6284 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087073    6284 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087075    6284 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087078    6284 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087080    6284 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087083    6284 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087087    6284 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087089    6284 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087092    6284 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:44:28.106102 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087094    6284 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087097    6284 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087100    6284 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087102    6284 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087105    6284 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087109    6284 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087112    6284 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087114    6284 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087117    6284 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087119    6284 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087123    6284 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087126    6284 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087129    6284 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:28.087131    6284 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.087197    6284 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.088308    6284 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.088388    6284 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.088582    6284 server.go:1014] "Starting client certificate rotation"
Nov 02 13:44:28.106863 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.088716    6284 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:44:28.107334 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.088791    6284 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:38:08 +0000 UTC" deadline="2028-04-02 01:29:38.012172637 +0000 UTC"
Nov 02 13:44:28.107334 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.088833    6284 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12395h45m9.923342501s"
Nov 02 13:44:28.107334 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.089180    6284 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:44:28.107334 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.090592    6284 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:44:28.107334 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.092172    6284 log.go:25] "Validated CRI v1 runtime API"
Nov 02 13:44:28.107334 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.094181    6284 log.go:25] "Validated CRI v1 image API"
Nov 02 13:44:28.107334 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.095022    6284 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 13:44:28.107334 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.098764    6284 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 99574455-4c74-4072-ad76-6ccfdf42f2e1:/dev/nvme0n1p3 9eda1afd-a622-451f-94f2-aa91a5e7518e:/dev/nvme0n1p4]
Nov 02 13:44:28.107584 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.098794    6284 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/050b71e53ba9388f7f10491ddf8541792adb9ee3ee2dda1ca442951deb1670cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/050b71e53ba9388f7f10491ddf8541792adb9ee3ee2dda1ca442951deb1670cf/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11783e1bcf6f17ce82b0adedfa657e45f78a89daebd7e34771e77c1bef5659ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11783e1bcf6f17ce82b0adedfa657e45f78a89daebd7e34771e77c1bef5659ee/userdata/shm major:0 minor:426 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/293dddafbc0ac9b18beefb2d7c6b1ae612e0a25d3f68aaa89a0f5671c5ca083b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/293dddafbc0ac9b18beefb2d7c6b1ae612e0a25d3f68aaa89a0f5671c5ca083b/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32a102b0508db4ce09b08788497e2387b3e6b14f2cd8dd0a0dd5b950eff44736/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32a102b0508db4ce09b08788497e2387b3e6b14f2cd8dd0a0dd5b950eff44736/userdata/shm major:0 minor:250 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fade1d8ef298325be12da5e63e05078f7fa8f4c9d45e3649e4b6bdd86fc8374/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fade1d8ef298325be12da5e63e05078f7fa8f4c9d45e3649e4b6bdd86fc8374/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/536d7e13a4d6066e9d1b3f035fb781c9209ba1564aa8244a9290c820fb4e8c74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/536d7e13a4d6066e9d1b3f035fb781c9209ba1564aa8244a9290c820fb4e8c74/userdata/shm major:0 minor:103 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57e708d0e6968e5778e6c8d9fc57a25dafc54276a08d45046aceffad61bc9bbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57e708d0e6968e5778e6c8d9fc57a25dafc54276a08d45046aceffad61bc9bbc/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5954afd4612bdd401080fe703e06ba03b4c27cd8f5bee438c63058155fe78e2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5954afd4612bdd401080fe703e06ba03b4c27cd8f5bee438c63058155fe78e2d/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/618b183d511f453e7c50efb4daa7077edd613cd91938fe7db120f64c3fef9f9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/618b183d511f453e7c50efb4daa7077edd613cd91938fe7db120f64c3fef9f9b/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/741a99eacd95e3aecc7c4e7631b2e615e502928b6e58c8abe3d4c76e2a6cb379/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/741a99eacd95e3aecc7c4e7631b2e615e502928b6e58c8abe3d4c76e2a6cb379/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/894c6eb722903e8f2cb1d6f76d661a42268a0af22297a52d690c7d123178f1ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/894c6eb722903e8f2cb1d6f76d661a42268a0af22297a52d690c7d123178f1ec/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/922aaf5edd9aa62864ffa50b6185f0e8166a495caa6979645c07bb5cc06414d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/922aaf5edd9aa62864ffa50b6185f0e8166a495caa6979645c07bb5cc06414d3/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b2ed9769382472c01d45016c359d28f9c73ca0b93ec11ff8681d526316a90f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b2ed9769382472c01d45016c359d28f9c73ca0b93ec11ff8681d526316a90f2/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fd06223f5e9f1f30d521b3ccae9f526baf46ea53418b559eca50d7dabc9623f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fd06223f5e9f1f30d521b3ccae9f526baf46ea53418b559eca50d7dabc9623f/userdata/shm major:0 minor:55 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bba513efe642ce0839e826b411cb03d094ba2fd4c1308e0758e196d6abfad6e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bba513efe642ce0839e826b411cb03d094ba2fd4c1308e0758e196d6abfad6e7/userdata/shm major:0 minor:424 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6b8b6f9d33ef0223dfa8aed556326d8065e19523786af4d8218f089e61ab414/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6b8b6f9d33ef0223dfa8aed556326d8065e19523786af4d8218f089e61ab414/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2b0f73d19c2fa0514d6a23b654b88058c0e04a50f1af46d30f711cbbcc2277f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2b0f73d19c2fa0514d6a23b654b88058c0e04a50f1af46d30f711cbbcc2277f/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5475d929c3a5746a5d83f5e066c55dc74590c80a87b3b094717def719f9e3fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5475d929c3a5746a5d83f5e066c55dc74590c80a87b3b094717def719f9e3fa/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de4fef001fb2f5113dd2fffac2d210041a30b797d5ea7e62bc921410bdc4ac02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de4fef001fb2f5113dd2fffac2d210041a30b797d5ea7e62bc921410bdc4ac02/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3180528833f71f0fcf8ffcd0c204bb3c10ebfc60236bc48c4387db0de0cb46e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3180528833f71f0fcf8ffcd0c204bb3c10ebfc60236bc48c4387db0de0cb46e/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eee4840a3e24b7d7fa57f555724ac0ee49a296dc595ca316657ce3d202d72c3f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eee4840a3e24b7d7fa57f555724ac0ee49a296dc595ca316657ce3d202d72c3f/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fed22b7ba9098af776862ae593258d6a4982a7d8192b859fa9f92354edd721ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fed22b7ba9098af776862ae593258d6a4982a7d8192b859fa9f92354edd721ae/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/094d4ae0-f6f8-44ae-a658-03f6322a3223/volumes/kubernetes.io~projected/kube-api-access-8nfr8:{mountpoint:/var/lib/kubelet/pods/094d4ae0-f6f8-44ae-a658-03f6322a3223/volumes/kubernetes.io~projected/kube-api-access-8nfr8 major:0 minor:77 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25220e90-f330-4bd6-b2ff-45eb20b344c2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/25220e90-f330-4bd6-b2ff-45eb20b344c2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25220e90-f330-4bd6-b2ff-45eb20b344c2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/25220e90-f330-4bd6-b2ff-45eb20b344c2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25220e90-f330-4bd6-b2ff-45eb20b344c2/volumes/kubernetes.io~projected/kube-api-access-wdgft:{mountpoint:/var/lib/kubelet/pods/25220e90-f330-4bd6-b2ff-45eb20b344c2/volumes/kubernetes.io~projected/kube-api-access-wdgft major:0 minor:76 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb/volumes/kubernetes.io~projected/kube-api-access-vdsdb:{mountpoint:/var/lib/kubelet/pods/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb/volumes/kubernetes.io~projected/kube-api-access-vdsdb major:0 minor:75 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35941a8e-d549-436f-b3e1-0be3ef2cc3a7/volumes/kubernetes.io~projected/kube-api-access-cs9jp:{mountpoint:/var/lib/kubelet/pods/35941a8e-d549-436f-b3e1-0be3ef2cc3a7/volumes/kubernetes.io~projected/kube-api-access-cs9jp major:0 minor:71 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/380def42-3dee-46c1-83b8-52910cc74811/volumes/kubernetes.io~projected/kube-api-access-srg7z:{mountpoint:/var/lib/kubelet/pods/380def42-3dee-46c1-83b8-52910cc74811/volumes/kubernetes.io~projected/kube-api-access-srg7z major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/380def42-3dee-46c1-83b8-52910cc74811/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/380def42-3dee-46c1-83b8-52910cc74811/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/380def42-3dee-46c1-83b8-52910cc74811/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/380def42-3dee-46c1-83b8-52910cc74811/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528f5273-2e64-4225-b59b-1616674a15fa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/528f5273-2e64-4225-b59b-1616674a15fa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528f5273-2e64-4225-b59b-1616674a15fa/volumes/kubernetes.io~projected/kube-api-access-xrmv6:{mountpoint:/var/lib/kubelet/pods/528f5273-2e64-4225-b59b-1616674a15fa/volumes/kubernetes.io~projected/kube-api-access-xrmv6 major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/528f5273-2e64-4225-b59b-1616674a15fa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/528f5273-2e64-4225-b59b-1616674a15fa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55837b60-e43f-4eb2-9505-c755bde739ec/volumes/kubernetes.io~projected/kube-api-access-fjkg7:{mountpoint:/var/lib/kubelet/pods/55837b60-e43f-4eb2-9505-c755bde739ec/volumes/kubernetes.io~projected/kube-api-access-fjkg7 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55837b60-e43f-4eb2-9505-c755bde739ec/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/55837b60-e43f-4eb2-9505-c755bde739ec/volumes/kubernetes.io~secret/cert major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/570c7233-f4c2-4c16-b9d9-67baa3fa7b96/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/570c7233-f4c2-4c16-b9d9-67baa3fa7b96/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ca4634-5648-402b-aaef-13f8d4ee2b4b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/72ca4634-5648-402b-aaef-13f8d4ee2b4b/volumes/kubernetes.io~secret/agent-certs major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~projected/kube-api-access-q9222:{mountpoint:/var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~projected/kube-api-access-q9222 major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e76a365-0c53-4369-b49e-b19a6611a391/volumes/kubernetes.io~projected/kube-api-access-bx5rq:{mountpoint:/var/lib/kubelet/pods/8e76a365-0c53-4369-b49e-b19a6611a391/volumes/kubernetes.io~projected/kube-api-access-bx5rq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e76a365-0c53-4369-b49e-b19a6611a391/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/8e76a365-0c53-4369-b49e-b19a6611a391/volumes/kubernetes.io~secret/metrics-tls major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/web-config/alertmanager/9 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/web-config/config-reloader/6 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~empty-dir/config-out major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~projected/kube-api-access-vrckt:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~projected/kube-api-access-vrckt major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~projected/tls-assets major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/config-volume major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/web-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99d52cb6-4109-40d5-8fc7-fb6679a03d60/volumes/kubernetes.io~projected/kube-api-access-mtmnw:{mountpoint:/var/lib/kubelet/pods/99d52cb6-4109-40d5-8fc7-fb6679a03d60/volumes/kubernetes.io~projected/kube-api-access-mtmnw major:0 minor:73 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df3afdd-f643-4485-a85d-f78211c7b169/volumes/kubernetes.io~projected/kube-api-access-g7dkm:{mountpoint:/var/lib/kubelet/pods/9df3afdd-f643-4485-a85d-f78211c7b169/volumes/kubernetes.io~projected/kube-api-access-g7dkm major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4/volumes/kubernetes.io~projected/kube-api-access-6q6b5:{mountpoint:/var/lib/kubelet/pods/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4/volumes/kubernetes.io~projected/kube-api-access-6q6b5 major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0175cfb-9276-42ed-bff8-f359a5727eb7/volumes/kubernetes.io~projected/kube-api-access-56w8p:{mountpoint:/var/lib/kubelet/pods/b0175cfb-9276-42ed-bff8-f359a5727eb7/volumes/kubernetes.io~projected/kube-api-access-56w8p major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0175cfb-9276-42ed-bff8-f359a5727eb7/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b0175cfb-9276-42ed-bff8-f359a5727eb7/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0175cfb-9276-42ed-bff8-f359a5727eb7/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/b0175cfb-9276-42ed-bff8-f359a5727eb7/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c640c634-2704-48e7-a121-e0975ae43e23/volumes/kubernetes.io~projected/kube-api-access-v9qcz:{mountpoint:/var/lib/kubelet/pods/c640c634-2704-48e7-a121-e0975ae43e23/volumes/kubernetes.io~projected/kube-api-access-v9qcz major:0 minor:72 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cff7b24a-4b98-4592-9344-5ec4930c6079/volumes/kubernetes.io~projected/kube-api-access-jbffg:{mountpoint:/var/lib/kubelet/pods/cff7b24a-4b98-4592-9344-5ec4930c6079/volumes/kubernetes.io~projected/kube-api-access-jbffg major:0 minor:74 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cff7b24a-4b98-4592-9344-5ec4930c6079/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/cff7b24a-4b98-4592-9344-5ec4930c6079/volumes/kubernetes.io~secret/metrics-certs major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volume-subpaths/web-config/config-reloader/2 major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volume-subpaths/web-config/prometheus/15 major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~empty-dir/config-out major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~projected/kube-api-access-bjndc:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~projected/kube-api-access-bjndc major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~projected/tls-assets major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/config major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/web-config major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6c927e4-b288-490a-acb9-c8e61846da1e/volumes/kubernetes.io~projected/kube-api-access-c5x76:{mountpoint:/var/lib/kubelet/pods/e6c927e4-b288-490a-acb9-c8e61846da1e/volumes/kubernetes.io~projected/kube-api-access-c5x76 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6c927e4-b288-490a-acb9-c8e61846da1e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e6c927e4-b288-490a-acb9-c8e61846da1e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad8eb6b-fa80-41f6-b5ff-51c39bba446c/volumes/kubernetes.io~projected/kube-api-access-cczzh:{mountpoint:/var/lib/kubelet/pods/fad8eb6b-fa80-41f6-b5ff-51c39bba446c/volumes/kubernetes.io~projected/kube-api-access-cczzh major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad8eb6b-fa80-41f6-b5ff-51c39bba446c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fad8eb6b-fa80-41f6-b5ff-51c39bba446c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fad8eb6b-fa80-41f6-b5ff-51c39bba446c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/fad8eb6b-fa80-41f6-b5ff-51c39bba446c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:246 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-101:{mountpoint:/var/lib/containers/storage/overlay/2e424216f71ec7cff9af516feb2bacec6d677c71ed51de1d1121fee5417c0ade/merged major:0 minor:101 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/a122c249954e0759fd4c4215dffa0dec56268193407eafdc64fefb398eb852c4/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/d65b62cd38a08f9fa613d43a2ca6fba6e1588314541d75bd173761d66895b9e9/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/15533e0c9782d26475c2f722fdb8153fdcf2a32e743d43623dee202e4df52a37/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/09a52ba3fecabcd57784a894454903ae1be67e8fa19b86e158d590414c5de8f6/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/80233d3c3a227a01adeff475a770cd4d1eabf9559271076ca50472e68676d0cf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/959ce8e1cf2a3342eaa9f9541cf516993e0facbd239f4f500cd215991eedd463/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ae5213dfb761d3f1f6f6bb36429b6161a890c86ce7d096579ae8d719ec5a442e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/12eec290eb3266d3fbac21526980f5d85f80f4eab1c3f54341bd2b8af0913497/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/44549b7bad3e0797b7f4e5975efd893d96569cd454b3700bb310cd2ef6ce1b26/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-150:{mountpoint:/var/lib/containers/storage/overlay/bbdadca0c45d5c5cec88c849b2653ccb93b177cf387c87f059f4c824173a6983/merged major:0 minor:150 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7fe0e29d891dcc1ae0c27e1c501955813441acab42080857e019e9918d61b85f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/aed3102c7deed13eac76ad3e5fa44bde5146f0a93326e48110628998cf6f3eb7/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/dcc06bca1ec6df1bcdef47edaba4bf19735e377953f0feb41bc2ba38a026b759/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/9eb527b541dab6fd46a9d2322dd31251997d086129df78b4eee74f7a84f1ef54/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/d3f54be1674d6484b1f0cb202f08abf30e1b7c4c00f7da64ce1761f1c8b355a3/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/667546d663ef33511749385b96b71606a5ee31b6f7da30322b97d92bad05f414/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/c6bf91f9f3b4f693029955c0c655e10ec72d6e796f3e5fa225eedd60574dd787/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/3e0802a5fab3d46810cbade4a76974c15c846e3186761e1d72b00a7e817f006f/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/6a19e636f725d3d1fb8304220e68b9b719036af41c6540aaad20652a4d0e79f3/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f76da20f50ed9a1b072faaff6f82df4bb8617c54af8801cd0555e8c2df43ef19/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/74dd58466086024890ee40cdd15c38579e288e4390370c7fc190d4718b9bbaad/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/61c9c82517d37bce44fe963435ede454e93432fac833bbec830dc511f43d27b3/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/96798f9f696ceaf6885032a38a0c09bd53cfb540338fc96b84e98573b3deb88d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/d5e8656e7ff08c36967d7a273f3d5308c7c6b441d5452e47d464342a41b47a5f/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/85ac411ca6b22fc7eabf34467b28fc63bc03001012d16c150407aff8699cca80/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/ca90985d4086848001fa3550f807d016a9ac633d0e37d841e1ddc41985ade6ee/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/747d49e7328e1f54412fe3f38fad68323ce8481a5d195eafc8d3c04f102b6cb7/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/fad8280bbc43a192d1ed5e2d7d8e853a40210c37265a585572c71d54ee065dd6/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/bd5db6800dd14e6be74ee2011ec1821acbfaa4317116e5ed51966c7b44cdbdea/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/3eeccf458e308c9595dfd9286fbb726caad60c27b35bd1c34c224be9d20d5219/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/fc1bb47942e46e6f75b1a1a30e1edb8de6dd9a796d4417bfac7e6adfd0dced62/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/54e22dfa936cafa173941cb8f51e0507f95bbbd260f0d65fcd47fbe5f4df2191/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/2dc054189c391884abf2a857485f01bf107cd4e01051e941829d5eed415bb374/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/d2088b6e6d926a8f8a09319a467c7eefc57d9511efc9237c2f9dc14e60d8db7e/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/5411b127269a7767caacd6538dd4e6a93e0ff1ce74d31b202b1675c4cb0270fd/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/b1ee4a4a30d0ca9e86de8241c4f7cb6553f7e6026e6b1943da0d933a4b687b6b/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/9625f27dfc25b061ca4c9430c026dc865b5ad1ee12e95f927423fe657db83c65/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/0c28e253181f95b6f6792d78e3de0f433c999058f106da235d95ac8a5400b192/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/ae27baecb32cbed51ac77495d0de8ebc1c10759774d171d7e3fa60a64306f800/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/976ceb19a60e71366be5ae03115442a9b41c4964b4aab98c684ad51d2b7b7998/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/c8ddc9d806c0bb234aee5f7af3181a5da92980710ddf166c68042af73a7a075a/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/d2e6b1a2f1ccde06f403068962ca35008f363de22ad7757a2e6aa24918ed4516/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/d6b8b8121feb26073706fc188cc5a676b4f25888812ba674c98810171c9bf66b/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/9768a53267168bcf38a488b231beae8ada6c724587cb8ceac2cd8cbb8f4b1795/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/176a4ce73fa05f0d8ef2d0b476a3821180095156e1d243f3b0b8c82c4f6b1273/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/25f33fba5d885997352bf88cf5d78a689804c9aa9467ff8c95190a8081d9a595/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/041fb1c5b50d79047c2255d602c13afc3eea65d2ec5a6cee526cef0fcf8b55b6/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/08fe57fab45c944c7a95d97055b465b805efa563bb006b22ed8ac93e9a02d714/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/b6947ba794189f915b0fadbcdaa6a0b1fe2d106ed52537cacd544d00a3be6e0f/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/1df986940937ac2c3cc63666d5358224d1e9946382765497a16548b459acc5a1/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/af11f073cd36ac7acc6c68cc9e12480472662bb767b2bb4367758ba4e299b15b/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/f629753741032cf86573fd60ccef2e2464bc00e1145dc0eee4c16178c9292a5a/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/5d2020402bdba9a0562715a50c4390aac73a14d6d2e0a112687126bf9560c25f/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/8444a0c9a6f6b3081055ca14b7de751595e6accd6cf3ecaeaf24248d05be7577/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/ac7e3986a247330d1c0f9f5ed1195641a430edf33e6fd7a2bafc9cf25e08a681/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/88412ec7ad297c3d77cc991959be101d5a3ea75f704c9642c6627b979d922de9/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/69a1a2103c92b25712cfed4912c2fb60538183565c94b99edfa21ede04327621/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/608efd24de9278d31b26c54a93b305b210ae00e912e693bb9e227e68f1d2d2f4/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/56da1d46c328923d843f8ec5ded252d532b42e03a7fd9e269fc4fdb00e2f6afd/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/4cea0b09d6aa7da132c32043aa25908a0a48c49a34a39feaaa5773232e61043e/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/fb068c51150211ce908def1e0afe61a59670a76f489d109fc56d5f37f36b42f4/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/905a0dce7c205dc7d9f490f59fa7d5a7fd8fa1b7225d2828f8d8382765ec5cb3/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/c21ab163a76d1972ace34e961e706b37ccfb81590fc33163b3b2db54c17ed1a8/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/c6c9294bc3c1efa09dfd636b9bbd6aa04ad91fccbb0390b195dcb918f9a562f6/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/62b488e8fa275b6d875ff237545e483957488162347f79d5a4f78c53e8c433e7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/var/lib/containers/storage/overlay/8dc844e5d83e12ca9e2721dcb0fcc37207ccfd0efd4fb6938fa08913e92fd4a6/merged major:0 minor:57 fsType:overlay blockSize:0} overlay_0-60:{mountpoint:/var/lib/containers/storage/overlay/4801afc31fff740afdfdb3a9c63a9c8971b62d5a201b39ac7fb2a01dbd4fa3c6/merged major:0 minor:60 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/05a85980977d25263495621892dd49fba1916b0f80cfa9d9169cf49b26068618/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/8424ca6779e848a8609fbfedfd819f273f6152ca5fc7a8de97a1e6dfe93745ca/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/a503c724aaf9c21d58a21988939b6bb0899b6fe09e8f42dd12ee3765401436f2/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/032605e5c930e62dd46e238ebdd9d3a036e69915a6f9f2832405874882c61497/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/1d2da5e8bb9c12e3a190b9506577758c69b29b9c5aee3784dff5c52f9a997ad8/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-99:{mountpoint:/var/lib/containers/storage/overlay/4405adc82edd92d261cbd1abf86dcd6cc22526fed821a509e40a507c43cd37c5/merged major:0 minor:99 fsType:overlay blockSize:0}]
Nov 02 13:44:28.109836 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.109382    6284 manager.go:217] Machine: {Timestamp:2026-11-02 13:44:28.108864263 +0000 UTC m=+0.068239862 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109203 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cd6418663bef73cf6076002ca7a25 SystemUUID:ec2cd641-8663-bef7-3cf6-076002ca7a25 BootID:3d677981-2928-41b1-ab67-2bbd40e508da Filesystems:[{Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/050b71e53ba9388f7f10491ddf8541792adb9ee3ee2dda1ca442951deb1670cf/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/618b183d511f453e7c50efb4daa7077edd613cd91938fe7db120f64c3fef9f9b/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:328 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:368 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/528f5273-2e64-4225-b59b-1616674a15fa/volumes/kubernetes.io~projected/kube-api-access-xrmv6 DeviceMajor:0 DeviceMinor:69 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/380def42-3dee-46c1-83b8-52910cc74811/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cff7b24a-4b98-4592-9344-5ec4930c6079/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:416 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/25220e90-f330-4bd6-b2ff-45eb20b344c2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57e708d0e6968e5778e6c8d9fc57a25dafc54276a08d45046aceffad61bc9bbc/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/528f5273-2e64-4225-b59b-1616674a15fa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eee4840a3e24b7d7fa57f555724ac0ee49a296dc595ca316657ce3d202d72c3f/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4/volumes/kubernetes.io~projected/kube-api-access-6q6b5 DeviceMajor:0 DeviceMinor:70 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/380def42-3dee-46c1-83b8-52910cc74811/volumes/kubernetes.io~projected/kube-api-access-srg7z DeviceMajor:0 DeviceMinor:241 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~projected/kube-api-access-bjndc DeviceMajor:0 DeviceMinor:383 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2b0f73d19c2fa0514d6a23b654b88058c0e04a50f1af46d30f711cbbcc2277f/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25220e90-f330-4bd6-b2ff-45eb20b344c2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32a102b0508db4ce09b08788497e2387b3e6b14f2cd8dd0a0dd5b950eff44736/userdata/shm DeviceMajor:0 DeviceMinor:250 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:373 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:325 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:290 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c640c634-2704-48e7-a121-e0975ae43e23/volumes/kubernetes.io~projected/kube-api-access-v9qcz DeviceMajor:0 DeviceMinor:72 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6c927e4-b288-490a-acb9-c8e61846da1e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9df3afdd-f643-4485-a85d-f78211c7b169/volumes/kubernetes.io~projected/kube-api-access-g7dkm DeviceMajor:0 DeviceMinor:423 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-99 DeviceMajor:0 DeviceMinor:99 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fad8eb6b-fa80-41f6-b5ff-51c39bba446c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/293dddafbc0ac9b18beefb2d7c6b1ae612e0a25d3f68aaa89a0f5671c5ca083b/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/570c7233-f4c2-4c16-b9d9-67baa3fa7b96/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:420 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cff7b24a-4b98-4592-9344-5ec4930c6079/volumes/kubernetes.io~projected/kube-api-access-jbffg DeviceMajor:0 DeviceMinor:74 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e76a365-0c53-4369-b49e-b19a6611a391/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:200 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b0175cfb-9276-42ed-bff8-f359a5727eb7/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:245 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:380 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de4fef001fb2f5113dd2fffac2d210041a30b797d5ea7e62bc921410bdc4ac02/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-101 DeviceMajor:0 DeviceMinor:101 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5954afd4612bdd401080fe703e06ba03b4c27cd8f5bee438c63058155fe78e2d/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b2ed9769382472c01d45016c359d28f9c73ca0b93ec11ff8681d526316a90f2/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bba513efe642ce0839e826b411cb03d094ba2fd4c1308e0758e196d6abfad6e7/userdata/shm DeviceMajor:0 DeviceMinor:424 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb/volumes/kubernetes.io~projected/kube-api-access-vdsdb DeviceMajor:0 DeviceMinor:75 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6c927e4-b288-490a-acb9-c8e61846da1e/volumes/kubernetes.io~projected/kube-api-access-c5x76 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/25220e90-f330-4bd6-b2ff-45eb20b344c2/volumes/kubernetes.io~projected/kube-api-access-wdgft DeviceMajor:0 DeviceMinor:76 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~projected/kube-api-access-q9222 DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6b8b6f9d33ef0223dfa8aed556326d8065e19523786af4d8218f089e61ab414/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:374 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:378 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:379 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:375 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/922aaf5edd9aa62864ffa50b6185f0e8166a495caa6979645c07bb5cc06414d3/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b0175cfb-9276-42ed-bff8-f359a5727eb7/volumes/kubernetes.io~projected/kube-api-access-56w8p DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:375 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-60 DeviceMajor:0 DeviceMinor:60 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11783e1bcf6f17ce82b0adedfa657e45f78a89daebd7e34771e77c1bef5659ee/userdata/shm DeviceMajor:0 DeviceMinor:426 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/35941a8e-d549-436f-b3e1-0be3ef2cc3a7/volumes/kubernetes.io~projected/kube-api-access-cs9jp DeviceMajor:0 DeviceMinor:71 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fd06223f5e9f1f30d521b3ccae9f526baf46ea53418b559eca50d7dabc9623f/userdata/shm DeviceMajor:0 DeviceMinor:55 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0175cfb-9276-42ed-bff8-f359a5727eb7/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:247 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ca4634-5648-402b-aaef-13f8d4ee2b4b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/528f5273-2e64-4225-b59b-1616674a15fa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:68 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5475d929c3a5746a5d83f5e066c55dc74590c80a87b3b094717def719f9e3fa/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99d52cb6-4109-40d5-8fc7-fb6679a03d60/volumes/kubernetes.io~projected/kube-api-access-mtmnw DeviceMajor:0 DeviceMinor:73 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fad8eb6b-fa80-41f6-b5ff-51c39bba446c/volumes/kubernetes.io~projected/kube-api-access-cczzh DeviceMajor:0 DeviceMinor:249 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-150 DeviceMajor:0 DeviceMinor:150 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~projected/kube-api-access-vrckt DeviceMajor:0 DeviceMinor:291 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/894c6eb722903e8f2cb1d6f76d661a42268a0af22297a52d690c7d123178f1ec/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/094d4ae0-f6f8-44ae-a658-03f6322a3223/volumes/kubernetes.io~projected/kube-api-access-8nfr8 DeviceMajor:0 DeviceMinor:77 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/380def42-3dee-46c1-83b8-52910cc74811/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:236 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fade1d8ef298325be12da5e63e05078f7fa8f4c9d45e3649e4b6bdd86fc8374/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/536d7e13a4d6066e9d1b3f035fb781c9209ba1564aa8244a9290c820fb4e8c74/userdata/shm DeviceMajor:0 DeviceMinor:103 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55837b60-e43f-4eb2-9505-c755bde739ec/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:196 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/55837b60-e43f-4eb2-9505-c755bde739ec/volumes/kubernetes.io~projected/kube-api-access-fjkg7 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fad8eb6b-fa80-41f6-b5ff-51c39bba446c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:244 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:377 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:382 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/741a99eacd95e3aecc7c4e7631b2e615e502928b6e58c8abe3d4c76e2a6cb379/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e76a365-0c53-4369-b49e-b19a6611a391/volumes/kubernetes.io~projected/kube-api-access-bx5rq DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:375 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fed22b7ba9098af776862ae593258d6a4982a7d8192b859fa9f92354edd721ae/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3180528833f71f0fcf8ffcd0c204bb3c10ebfc60236bc48c4387db0de0cb46e/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:375 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7dc2b96a-b979-4de1-a69b-ce1c566518c6/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:376 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:11783e1bcf6f17c MacAddress:ae:c3:c0:c5:ef:61 Speed:10000 Mtu:8901} {Name:293dddafbc0ac9b MacAddress:5e:21:89:a6:d8:74 Speed:10000 Mtu:8901} {Name:32a102b0508db4c MacAddress:76:64:16:9c:c2:33 Speed:10000 Mtu:8901} {Name:5954afd4612bdd4 MacAddress:fe:dc:44:1d:28:24 Speed:10000 Mtu:8901} {Name:894c6eb722903e8 MacAddress:2a:78:25:a1:aa:bc Speed:10000 Mtu:8901} {Name:922aaf5edd9aa62 MacAddress:56:50:e4:5d:ce:d2 Speed:10000 Mtu:8901} {Name:9b2ed9769382472 MacAddress:5e:ba:aa:e7:a3:14 Speed:10000 Mtu:8901} {Name:bba513efe642ce0 MacAddress:be:cc:fd:1a:cb:66 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c1:41:bc:c7 Speed:0 Mtu:9001} {Name:br-int MacAddress:c6:9e:a3:e6:59:db Speed:0 Mtu:8901} {Name:e3180528833f71f MacAddress:4a:61:86:78:45:7d Speed:10000 Mtu:8901} {Name:eee4840a3e24b7d MacAddress:fe:fb:37:9c:9b:7d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c1:41:bc:c7 Speed:0 Mtu:9001} {Name:fed22b7ba9098af MacAddress:1e:57:de:30:5d:fd Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:12:54:30:38:db:48 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5a:4f:1d:7f:3a:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 13:44:28.110100 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.109841    6284 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 13:44:28.110100 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.109909    6284 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 13:44:28.110315 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110285    6284 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 13:44:28.110459 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110316    6284 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 13:44:28.110501 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110462    6284 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 13:44:28.110501 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110470    6284 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 13:44:28.110501 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110481    6284 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:44:28.110501 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110496    6284 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:44:28.110619 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110611    6284 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 13:44:28.110809 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110799    6284 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 13:44:28.110921 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110912    6284 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 13:44:28.111045 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110968    6284 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 13:44:28.111045 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110979    6284 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 13:44:28.111045 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.110990    6284 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 13:44:28.111045 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.111000    6284 kubelet.go:405] "Adding apiserver pod source"
Nov 02 13:44:28.111045 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.111009    6284 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 13:44:28.111670 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.111655    6284 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 13:44:28.111670 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.111670    6284 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:44:28.112324 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.112302    6284 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 13:44:28.115827 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.115809    6284 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 13:44:28.115827 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.115828    6284 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:44:28.115963 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.115949    6284 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 13:44:28.116173 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116160    6284 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 13:44:28.116255 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116192    6284 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 13:44:28.116298 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116288    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 13:44:28.116344 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116303    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 13:44:28.116344 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116311    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 13:44:28.116344 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116319    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 13:44:28.116344 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116328    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 13:44:28.116344 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116335    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 13:44:28.116344 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116341    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 13:44:28.116344 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116347    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 13:44:28.116621 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116355    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 13:44:28.116621 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116361    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 13:44:28.116621 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116370    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 13:44:28.116621 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116388    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 13:44:28.116621 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116420    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 13:44:28.116621 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.116428    6284 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 13:44:28.120458 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.120039    6284 server.go:1263] "Started kubelet"
Nov 02 13:44:28.120458 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.120126    6284 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 13:44:28.120458 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.120231    6284 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 13:44:28.120679 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.120495    6284 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 13:44:28.120711 ip-10-0-131-98 systemd[1]: Started Kubernetes Kubelet.
Nov 02 13:44:28.121289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.121198    6284 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 13:44:28.122424 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.122404    6284 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 13:44:28.127821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.126906    6284 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 13:44:28.127821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.126950    6284 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 13:44:28.127821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.127028    6284 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:38:18 +0000 UTC" deadline="2028-04-01 09:11:47.997410206 +0000 UTC"
Nov 02 13:44:28.127821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.127047    6284 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 13:44:28.127821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.127063    6284 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 13:44:28.127821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.127155    6284 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 13:44:28.127821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.127051    6284 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12379h27m19.870362569s"
Nov 02 13:44:28.127821 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:28.127813    6284 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-98.ec2.internal\" not found"
Nov 02 13:44:28.129391 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.129368    6284 factory.go:55] Registering systemd factory
Nov 02 13:44:28.129391 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.129391    6284 factory.go:223] Registration of the systemd container factory successfully
Nov 02 13:44:28.129658 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.129645    6284 factory.go:153] Registering CRI-O factory
Nov 02 13:44:28.129717 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.129662    6284 factory.go:223] Registration of the crio container factory successfully
Nov 02 13:44:28.129768 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.129720    6284 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 13:44:28.129768 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.129743    6284 factory.go:103] Registering Raw factory
Nov 02 13:44:28.129768 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.129758    6284 manager.go:1199] Started watching for new ooms in manager
Nov 02 13:44:28.130088 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130045    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad8eb6b-fa80-41f6-b5ff-51c39bba446c" volumeName="kubernetes.io/empty-dir/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-textfile" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130092    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="094d4ae0-f6f8-44ae-a658-03f6322a3223" volumeName="kubernetes.io/projected/094d4ae0-f6f8-44ae-a658-03f6322a3223-kube-api-access-8nfr8" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130105    6284 manager.go:319] Starting recovery of all containers
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130109    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc2b96a-b979-4de1-a69b-ce1c566518c6" volumeName="kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130123    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130138    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0175cfb-9276-42ed-bff8-f359a5727eb7" volumeName="kubernetes.io/empty-dir/b0175cfb-9276-42ed-bff8-f359a5727eb7-volume-directive-shadow" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130151    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130165    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb" volumeName="kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-daemon-config" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130178    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb" volumeName="kubernetes.io/projected/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-kube-api-access-vdsdb" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130205    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380def42-3dee-46c1-83b8-52910cc74811" volumeName="kubernetes.io/configmap/380def42-3dee-46c1-83b8-52910cc74811-metrics-client-ca" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130220    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380def42-3dee-46c1-83b8-52910cc74811" volumeName="kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130234    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f5273-2e64-4225-b59b-1616674a15fa" volumeName="kubernetes.io/projected/528f5273-2e64-4225-b59b-1616674a15fa-kube-api-access-xrmv6" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130247    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ca4634-5648-402b-aaef-13f8d4ee2b4b" volumeName="kubernetes.io/configmap/72ca4634-5648-402b-aaef-13f8d4ee2b4b-konnectivity-ca" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130260    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc2b96a-b979-4de1-a69b-ce1c566518c6" volumeName="kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-client-ca-bundle" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130273    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e76a365-0c53-4369-b49e-b19a6611a391" volumeName="kubernetes.io/configmap/8e76a365-0c53-4369-b49e-b19a6611a391-config-volume" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130293    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25220e90-f330-4bd6-b2ff-45eb20b344c2" volumeName="kubernetes.io/projected/25220e90-f330-4bd6-b2ff-45eb20b344c2-kube-api-access-wdgft" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130307    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f5273-2e64-4225-b59b-1616674a15fa" volumeName="kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130321    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-metrics-client-ca" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130352    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130397    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-out" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130410    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-cluster-tls-config" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130424    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-volume" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130438    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99d52cb6-4109-40d5-8fc7-fb6679a03d60" volumeName="kubernetes.io/configmap/99d52cb6-4109-40d5-8fc7-fb6679a03d60-serviceca" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130453    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130467    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130480    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cff7b24a-4b98-4592-9344-5ec4930c6079" volumeName="kubernetes.io/projected/cff7b24a-4b98-4592-9344-5ec4930c6079-kube-api-access-jbffg" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130497    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130516    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130549    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130570    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35941a8e-d549-436f-b3e1-0be3ef2cc3a7" volumeName="kubernetes.io/empty-dir/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-tmp-dir" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130583    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0175cfb-9276-42ed-bff8-f359a5727eb7" volumeName="kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130597    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c640c634-2704-48e7-a121-e0975ae43e23" volumeName="kubernetes.io/configmap/c640c634-2704-48e7-a121-e0975ae43e23-iptables-alerter-script" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130610    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130624    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-kube-api-access-bjndc" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130640    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad8eb6b-fa80-41f6-b5ff-51c39bba446c" volumeName="kubernetes.io/projected/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-kube-api-access-cczzh" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130654    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" volumeName="kubernetes.io/projected/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-kube-api-access-6q6b5" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130671    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130686    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35941a8e-d549-436f-b3e1-0be3ef2cc3a7" volumeName="kubernetes.io/projected/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-kube-api-access-cs9jp" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130699    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc2b96a-b979-4de1-a69b-ce1c566518c6" volumeName="kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130712    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130725    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-config-out" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130741    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25220e90-f330-4bd6-b2ff-45eb20b344c2" volumeName="kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-tmp" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130755    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ca4634-5648-402b-aaef-13f8d4ee2b4b" volumeName="kubernetes.io/secret/72ca4634-5648-402b-aaef-13f8d4ee2b4b-agent-certs" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130767    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e76a365-0c53-4369-b49e-b19a6611a391" volumeName="kubernetes.io/empty-dir/8e76a365-0c53-4369-b49e-b19a6611a391-tmp-dir" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130780    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-tls-assets" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130795    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-web-config" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130809    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c640c634-2704-48e7-a121-e0975ae43e23" volumeName="kubernetes.io/projected/c640c634-2704-48e7-a121-e0975ae43e23-kube-api-access-v9qcz" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130822    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-tls-assets" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130835    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc2b96a-b979-4de1-a69b-ce1c566518c6" volumeName="kubernetes.io/empty-dir/7dc2b96a-b979-4de1-a69b-ce1c566518c6-audit-log" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130848    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df3afdd-f643-4485-a85d-f78211c7b169" volumeName="kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130863    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c927e4-b288-490a-acb9-c8e61846da1e" volumeName="kubernetes.io/empty-dir/e6c927e4-b288-490a-acb9-c8e61846da1e-data-volume" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130877    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad8eb6b-fa80-41f6-b5ff-51c39bba446c" volumeName="kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130890    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc2b96a-b979-4de1-a69b-ce1c566518c6" volumeName="kubernetes.io/projected/7dc2b96a-b979-4de1-a69b-ce1c566518c6-kube-api-access-q9222" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130903    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-main-db" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130919    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0175cfb-9276-42ed-bff8-f359a5727eb7" volumeName="kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130949    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c927e4-b288-490a-acb9-c8e61846da1e" volumeName="kubernetes.io/secret/e6c927e4-b288-490a-acb9-c8e61846da1e-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130962    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad8eb6b-fa80-41f6-b5ff-51c39bba446c" volumeName="kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-tls" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.130992    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380def42-3dee-46c1-83b8-52910cc74811" volumeName="kubernetes.io/projected/380def42-3dee-46c1-83b8-52910cc74811-kube-api-access-srg7z" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131005    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f5273-2e64-4225-b59b-1616674a15fa" volumeName="kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-env-overrides" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131017    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc2b96a-b979-4de1-a69b-ce1c566518c6" volumeName="kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131030    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" volumeName="kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131043    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cff7b24a-4b98-4592-9344-5ec4930c6079" volumeName="kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131056    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-db" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131074    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380def42-3dee-46c1-83b8-52910cc74811" volumeName="kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131087    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f5273-2e64-4225-b59b-1616674a15fa" volumeName="kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-config" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131102    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="570c7233-f4c2-4c16-b9d9-67baa3fa7b96" volumeName="kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131130    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e76a365-0c53-4369-b49e-b19a6611a391" volumeName="kubernetes.io/projected/8e76a365-0c53-4369-b49e-b19a6611a391-kube-api-access-bx5rq" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131146    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e76a365-0c53-4369-b49e-b19a6611a391" volumeName="kubernetes.io/secret/8e76a365-0c53-4369-b49e-b19a6611a391-metrics-tls" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131164    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131181    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131194    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131207    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb" volumeName="kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cni-binary-copy" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131235    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-kube-api-access-vrckt" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131249    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99d52cb6-4109-40d5-8fc7-fb6679a03d60" volumeName="kubernetes.io/projected/99d52cb6-4109-40d5-8fc7-fb6679a03d60-kube-api-access-mtmnw" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131263    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0175cfb-9276-42ed-bff8-f359a5727eb7" volumeName="kubernetes.io/projected/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-api-access-56w8p" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131277    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0175cfb-9276-42ed-bff8-f359a5727eb7" volumeName="kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-tls" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131290    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-web-config" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131303    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131315    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad8eb6b-fa80-41f6-b5ff-51c39bba446c" volumeName="kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-metrics-client-ca" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131328    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc2b96a-b979-4de1-a69b-ce1c566518c6" volumeName="kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-tls" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131341    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" volumeName="kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-binary-copy" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131355    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-config" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131369    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-grpc-tls" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131383    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25220e90-f330-4bd6-b2ff-45eb20b344c2" volumeName="kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-tuned" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131408    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55837b60-e43f-4eb2-9505-c755bde739ec" volumeName="kubernetes.io/projected/55837b60-e43f-4eb2-9505-c755bde739ec-kube-api-access-fjkg7" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131421    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" volumeName="kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131445    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0175cfb-9276-42ed-bff8-f359a5727eb7" volumeName="kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-metrics-client-ca" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131467    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-metrics-client-certs" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131512    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c927e4-b288-490a-acb9-c8e61846da1e" volumeName="kubernetes.io/configmap/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131521    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="528f5273-2e64-4225-b59b-1616674a15fa" volumeName="kubernetes.io/secret/528f5273-2e64-4225-b59b-1616674a15fa-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131546    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55837b60-e43f-4eb2-9505-c755bde739ec" volumeName="kubernetes.io/secret/55837b60-e43f-4eb2-9505-c755bde739ec-cert" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131562    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" volumeName="kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131570    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5bed06-f76d-474a-88ed-1fc6be553533" volumeName="kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131586    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c927e4-b288-490a-acb9-c8e61846da1e" volumeName="kubernetes.io/projected/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-api-access-c5x76" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131593    6284 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fad8eb6b-fa80-41f6-b5ff-51c39bba446c" volumeName="kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131613    6284 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 13:44:28.136085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.131623    6284 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 13:44:28.159791 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.159735    6284 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 13:44:28.163205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.161504    6284 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 13:44:28.163205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.161539    6284 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 13:44:28.163205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.161565    6284 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 13:44:28.163205 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:28.161615    6284 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 13:44:28.164721 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.164692    6284 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 13:44:28.165227 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.165202    6284 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:44:28.167079 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.167047    6284 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:44:28.176288 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.176252    6284 generic.go:356] "Generic (PLEG): container finished" podID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerID="e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741" exitCode=0
Nov 02 13:44:28.184498 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.184461    6284 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:44:28.184952 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.184915    6284 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:44:28.197272 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.197227    6284 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="d0091d56433015751224a7ce390ce21f36f1af62bd41de07017f6091b5a600db" exitCode=0
Nov 02 13:44:28.197272 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.197270    6284 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="07a6872ca12066354044b3a01b3b3b6bbfd4ab481c341c7f16b65b611d54ef15" exitCode=0
Nov 02 13:44:28.197483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.197283    6284 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="62d32104f3ace1ee342a972c733e744cc2435763720ee35ac4e0f5e80d6a76fa" exitCode=0
Nov 02 13:44:28.197483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.197294    6284 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="4460650740d3db7c959ae72e778f2250ace9db959063e479e2beab394c0f28b0" exitCode=0
Nov 02 13:44:28.197483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.197303    6284 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="b277846e1624087cff0bd5ffc9481ab561f7046ef784ea952b202d935235ed6e" exitCode=0
Nov 02 13:44:28.197483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.197311    6284 generic.go:356] "Generic (PLEG): container finished" podID="ac48fd07-ce48-4fc0-afa0-e9b31baaaec4" containerID="78aa95da86238feda4ef29da8bef3f6a2db2c0b629c58ad50581585b61d97f83" exitCode=0
Nov 02 13:44:28.205239 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.205208    6284 generic.go:356] "Generic (PLEG): container finished" podID="fad8eb6b-fa80-41f6-b5ff-51c39bba446c" containerID="5bf20c211a48dd042532c014bca001f27375681082c55d0dafb505be4d8173ac" exitCode=0
Nov 02 13:44:28.209089 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.209053    6284 generic.go:356] "Generic (PLEG): container finished" podID="da5bed06-f76d-474a-88ed-1fc6be553533" containerID="e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504" exitCode=0
Nov 02 13:44:28.227706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.227669    6284 generic.go:356] "Generic (PLEG): container finished" podID="d005c7e1a378e0971dc1c77f9a40f556" containerID="065fc2231ae9c9877bb03017332062bf802feae90307084906dbfd48f2bb358d" exitCode=0
Nov 02 13:44:28.249884 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.249849    6284 manager.go:324] Recovery completed
Nov 02 13:44:28.262388 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:28.262354    6284 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 13:44:28.276311 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276278    6284 cpu_manager.go:226] "Starting" policy="none"
Nov 02 13:44:28.276311 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276299    6284 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 13:44:28.276530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276324    6284 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 13:44:28.276530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276469    6284 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 13:44:28.276530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276480    6284 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 13:44:28.276530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276497    6284 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 13:44:28.276530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276502    6284 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 13:44:28.276530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276507    6284 policy_none.go:50] "Start"
Nov 02 13:44:28.276530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276517    6284 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 13:44:28.276530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276526    6284 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 13:44:28.276795 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276655    6284 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 13:44:28.276795 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276661    6284 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 13:44:28.276795 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.276667    6284 policy_none.go:44] "Start"
Nov 02 13:44:28.281597 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.281555    6284 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 13:44:28.284481 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.284464    6284 manager.go:341] "Starting Device Plugin manager"
Nov 02 13:44:28.285028 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:28.284493    6284 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 13:44:28.285109 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.285031    6284 server.go:87] "Starting device plugin registration server"
Nov 02 13:44:28.285320 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.285309    6284 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 13:44:28.285351 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.285322    6284 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 13:44:28.285460 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.285434    6284 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 13:44:28.285587 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.285554    6284 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 13:44:28.285821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.285602    6284 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 13:44:28.285821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.285811    6284 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 13:44:28.286019 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.285891    6284 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:44:28.285582534Z","UUID":"b2b14bfe-9a4f-4cda-882a-ec38bfc0987c","Handler":null,"Name":"","Endpoint":""}
Nov 02 13:44:28.287165 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:28.287143    6284 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 13:44:28.287371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.287354    6284 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 13:44:28.287441 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.287383    6284 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 13:44:28.386033 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.385997    6284 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:44:28.388194 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.388170    6284 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:44:28.388325 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.388207    6284 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:44:28.388325 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.388220    6284 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-98.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:44:28.388325 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.388266    6284 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.418146 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.418115    6284 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.418326 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.418207    6284 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.462903 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.462826    6284 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"]
Nov 02 13:44:28.463507 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.463468    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal" event={"ID":"41b5d311172faec6b74eed63c4c5b623","Type":"ContainerStarted","Data":"a16f90d288e49eafb73084350bd6380bc2b4ab7f5fe64e1051f0d2e72fb277a4"}
Nov 02 13:44:28.463576 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.463520    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal" event={"ID":"41b5d311172faec6b74eed63c4c5b623","Type":"ContainerStarted","Data":"741a99eacd95e3aecc7c4e7631b2e615e502928b6e58c8abe3d4c76e2a6cb379"}
Nov 02 13:44:28.463576 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.463529    6284 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.463729 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.463706    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal" event={"ID":"d005c7e1a378e0971dc1c77f9a40f556","Type":"ContainerStarted","Data":"a802a589d0528862b6df75d668167c7c962add104ad97257e4febe388f262ee3"}
Nov 02 13:44:28.463800 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.463736    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal" event={"ID":"d005c7e1a378e0971dc1c77f9a40f556","Type":"ContainerDied","Data":"065fc2231ae9c9877bb03017332062bf802feae90307084906dbfd48f2bb358d"}
Nov 02 13:44:28.463800 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.463757    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal" event={"ID":"d005c7e1a378e0971dc1c77f9a40f556","Type":"ContainerStarted","Data":"d2b0f73d19c2fa0514d6a23b654b88058c0e04a50f1af46d30f711cbbcc2277f"}
Nov 02 13:44:28.463901 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.463837    6284 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.484862 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.484825    6284 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:44:28.485031 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:28.484899    6284 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-98.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.485529 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.485513    6284 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:44:28.485573 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:28.485553    6284 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.567042 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.566948    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41b5d311172faec6b74eed63c4c5b623-config\") pod \"kube-apiserver-proxy-ip-10-0-131-98.ec2.internal\" (UID: \"41b5d311172faec6b74eed63c4c5b623\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.567042 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.566988    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.567219 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.567059    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.667641 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.667592    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41b5d311172faec6b74eed63c4c5b623-config\") pod \"kube-apiserver-proxy-ip-10-0-131-98.ec2.internal\" (UID: \"41b5d311172faec6b74eed63c4c5b623\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.667641 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.667642    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.667780 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.667659    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.667780 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.667710    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.667780 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.667718    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41b5d311172faec6b74eed63c4c5b623-config\") pod \"kube-apiserver-proxy-ip-10-0-131-98.ec2.internal\" (UID: \"41b5d311172faec6b74eed63c4c5b623\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:28.667780 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:28.667718    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d005c7e1a378e0971dc1c77f9a40f556-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" (UID: \"d005c7e1a378e0971dc1c77f9a40f556\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:29.111375 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.111328    6284 apiserver.go:52] "Watching apiserver"
Nov 02 13:44:29.145035 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.144996    6284 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 13:44:29.145755 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.145726    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-57d72","kube-system/kube-apiserver-proxy-ip-10-0-131-98.ec2.internal","openshift-network-operator/iptables-alerter-4j587","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k","openshift-multus/network-metrics-daemon-jcqf2","openshift-monitoring/node-exporter-jgpx4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xgnrc","openshift-monitoring/metrics-server-566f4c7487-mbcfx","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-cgnfh","kube-system/global-pull-secret-syncer-vsx66","openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c","openshift-ovn-kubernetes/ovnkube-node-sr6pw","openshift-dns/dns-default-lq2nc","openshift-monitoring/alertmanager-main-0","kube-system/konnectivity-agent-pd8nn","openshift-image-registry/node-ca-gkg2v","openshift-multus/multus-zf8tf","openshift-dns/node-resolver-5jbwx","openshift-insights/insights-runtime-extractor-lrzvt","openshift-ingress-canary/ingress-canary-n2d6j","openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"]
Nov 02 13:44:29.159319 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159289    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 13:44:29.159319 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159296    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5dqzs\""
Nov 02 13:44:29.159319 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159300    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qnz8k\""
Nov 02 13:44:29.159613 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159366    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-47zgd\""
Nov 02 13:44:29.159613 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159453    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 13:44:29.159711 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159610    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bqvhh\""
Nov 02 13:44:29.159711 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159624    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.159711 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159640    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.159711 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159685    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 13:44:29.159711 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159696    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.159876 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159783    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.159876 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159793    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 13:44:29.159876 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159797    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.159876 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159829    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 13:44:29.159876 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.159862    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 13:44:29.166561 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.166536    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-52vg6\""
Nov 02 13:44:29.166629 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.166573    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6lw86\""
Nov 02 13:44:29.168483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.168463    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 13:44:29.168588 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.168552    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.169274 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.169258    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 13:44:29.169927 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.169912    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.170030 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.169950    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.170030 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.169955    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.170030 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.169959    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 13:44:29.170136 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170034    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4wcws\""
Nov 02 13:44:29.170413 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170393    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 13:44:29.170413 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170417    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.170413 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170436    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lj29d\""
Nov 02 13:44:29.170657 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170487    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 13:44:29.170657 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170503    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xwmgj\""
Nov 02 13:44:29.170657 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170490    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.170657 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170530    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 13:44:29.170657 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170535    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.170657 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170559    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 13:44:29.170657 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170561    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lvdl4\""
Nov 02 13:44:29.170657 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170595    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.170883 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.170694    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.171068 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171055    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.171128 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171091    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.171202 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171187    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 13:44:29.171296 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171277    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 13:44:29.171328 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171278    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 13:44:29.171362 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171342    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hnbmg\""
Nov 02 13:44:29.171391 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171362    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 13:44:29.171439 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171401    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 13:44:29.171439 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171404    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.171439 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171433    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.171590 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171535    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 13:44:29.171590 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.171540    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w4l5x\""
Nov 02 13:44:29.172114 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.172102    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 13:44:29.172220 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.172182    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 13:44:29.172220 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.172209    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 13:44:29.172392 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.172379    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.172459 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.172383    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 13:44:29.172459 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.172396    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 13:44:29.172459 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.172426    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.172597 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.172585    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 13:44:29.180737 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180714    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 13:44:29.180737 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180716    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 13:44:29.180737 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180741    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 13:44:29.181009 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180754    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 13:44:29.181009 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180754    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 13:44:29.181009 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180796    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 13:44:29.181009 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180835    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 13:44:29.181009 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180840    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 13:44:29.181009 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180854    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 13:44:29.181009 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180869    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 13:44:29.181009 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180909    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 13:44:29.181009 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.180927    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 13:44:29.181315 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.181037    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 13:44:29.181315 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.181093    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 13:44:29.181315 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.181112    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 13:44:29.181315 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.181292    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 13:44:29.181315 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.181296    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 13:44:29.181315 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.181293    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 13:44:29.181546 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.181327    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 13:44:29.182277 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182260    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 13:44:29.182380 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182288    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 13:44:29.182380 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182291    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 13:44:29.182380 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182332    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 13:44:29.182380 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182332    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4d48r\""
Nov 02 13:44:29.182380 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182355    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5tjtq\""
Nov 02 13:44:29.182619 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182424    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rxjnr\""
Nov 02 13:44:29.182619 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182612    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fs4v4\""
Nov 02 13:44:29.182718 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182709    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 13:44:29.182765 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182731    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7our0gqq5qckd\""
Nov 02 13:44:29.182812 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182787    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xv47k\""
Nov 02 13:44:29.182812 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.182797    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9sg29\""
Nov 02 13:44:29.183663 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.183639    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 13:44:29.183817 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.183764    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 13:44:29.183952 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.183922    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 13:44:29.184042 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.183965    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6wff6\""
Nov 02 13:44:29.184042 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.184025    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-97ql1c6l88l52\""
Nov 02 13:44:29.184344 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.184329    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 13:44:29.184444 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.184423    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 13:44:29.184612 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.184587    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 13:44:29.184735 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.184720    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f4zv8\""
Nov 02 13:44:29.184881 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.184867    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hkz22\""
Nov 02 13:44:29.185948 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.185913    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 13:44:29.186664 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.186642    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 13:44:29.187115 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.187071    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 13:44:29.230038 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.229995    6284 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 13:44:29.235265 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.235236    6284 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:29.257592 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.257556    6284 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:44:29.257749 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:29.257616    6284 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-98.ec2.internal"
Nov 02 13:44:29.269987 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.269927    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:44:29.270168 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270001    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-conf\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.270168 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270029    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-host\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.270168 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270051    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e76a365-0c53-4369-b49e-b19a6611a391-config-volume\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:44:29.270168 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270075    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.270168 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270128    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-conf-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.270168 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270163    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-sys-fs\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.270404 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270198    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b0175cfb-9276-42ed-bff8-f359a5727eb7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.270404 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270230    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:44:29.270404 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270283    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-kubelet\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.270404 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270329    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.270404 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270356    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b0175cfb-9276-42ed-bff8-f359a5727eb7-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.270404 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270365    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-web-config\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.270404 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270397    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e76a365-0c53-4369-b49e-b19a6611a391-config-volume\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:44:29.270715 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270467    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.270715 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270529    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysconfig\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.270715 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270566    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-kubernetes\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.270715 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270594    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9jp\" (UniqueName: \"kubernetes.io/projected/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-kube-api-access-cs9jp\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:44:29.270715 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270595    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-web-config\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.270715 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270600    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.270715 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270635    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6c927e4-b288-490a-acb9-c8e61846da1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.271037 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270730    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.271037 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270867    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e6c927e4-b288-490a-acb9-c8e61846da1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.271037 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270888    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-textfile\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.271037 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270924    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-multus\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.271037 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270983    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.271037 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.270994    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.271037 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271011    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.271037 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271018    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-textfile\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.271037 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271038    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-registration-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.271356 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271064    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e76a365-0c53-4369-b49e-b19a6611a391-metrics-tls\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:44:29.271356 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271090    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-systemd-units\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.271356 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271116    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.271356 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271198    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:44:29.271356 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271227    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.271356 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271243    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.271356 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271287    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e76a365-0c53-4369-b49e-b19a6611a391-metrics-tls\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:44:29.271356 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271340    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56w8p\" (UniqueName: \"kubernetes.io/projected/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-api-access-56w8p\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271383    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/380def42-3dee-46c1-83b8-52910cc74811-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271387    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-netns\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271434    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-var-lib-kubelet\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271453    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-socket-dir-parent\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271470    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271510    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271530    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-etc-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271549    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-bin\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271566    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkg7\" (UniqueName: \"kubernetes.io/projected/55837b60-e43f-4eb2-9505-c755bde739ec-kube-api-access-fjkg7\") pod \"ingress-canary-n2d6j\" (UID: \"55837b60-e43f-4eb2-9505-c755bde739ec\") " pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271586    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271596    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.271686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271633    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271753    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271802    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-system-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271840    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c640c634-2704-48e7-a121-e0975ae43e23-host-slash\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271875    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271889    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271903    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-netns\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271911    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271961    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff7b24a-4b98-4592-9344-5ec4930c6079-metrics-certs\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271972    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.271965    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-socket-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272009    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-web-config\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272049    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e6c927e4-b288-490a-acb9-c8e61846da1e-crio-socket\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272047    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272077    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-client-ca-bundle\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272047    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272123    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d52cb6-4109-40d5-8fc7-fb6679a03d60-host\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272158    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-ovn\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.272181 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272186    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272230    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272319    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-tmp\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272345    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272377    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272389    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-tmp\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272400    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272319    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-client-ca-bundle\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272402    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-web-config\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272425    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272447    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272451    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-multus-certs\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272496    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272529    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-original-pull-secret\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272601    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272608    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-volume\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272603    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272663    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-daemon-config\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272672    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272624    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272726    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-accelerators-collector-config\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272752    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272760    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cnibin\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272810    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272824    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272831    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srg7z\" (UniqueName: \"kubernetes.io/projected/380def42-3dee-46c1-83b8-52910cc74811-kube-api-access-srg7z\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272889    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-modprobe-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272912    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-accelerators-collector-config\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272924    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272979    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.272970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.272991    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-daemon-config\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273010    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273037    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-tls\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273057    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7dc2b96a-b979-4de1-a69b-ce1c566518c6-audit-log\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273073    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9222\" (UniqueName: \"kubernetes.io/projected/7dc2b96a-b979-4de1-a69b-ce1c566518c6-kube-api-access-q9222\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273098    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273124    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273133    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273147    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsdb\" (UniqueName: \"kubernetes.io/projected/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-kube-api-access-vdsdb\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273162    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7dc2b96a-b979-4de1-a69b-ce1c566518c6-audit-log\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273178    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273235    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-tls\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273236    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273262    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273264    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtmnw\" (UniqueName: \"kubernetes.io/projected/99d52cb6-4109-40d5-8fc7-fb6679a03d60-kube-api-access-mtmnw\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273389    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-binary-copy\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273385    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-kubelet\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273438    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-env-overrides\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273461    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273469    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-script-lib\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273539    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-device-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273568    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-dbus\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273573    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-env-overrides\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273598    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-wtmp\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273641    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cni-binary-copy\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273665    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-script-lib\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273686    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273756    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273825    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cni-binary-copy\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273882    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjndc\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-kube-api-access-bjndc\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273928    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.273977    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfr8\" (UniqueName: \"kubernetes.io/projected/094d4ae0-f6f8-44ae-a658-03f6322a3223-kube-api-access-8nfr8\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274056    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cczzh\" (UniqueName: \"kubernetes.io/projected/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-kube-api-access-cczzh\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274187    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-hosts-file\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274199    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.274221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274218    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbffg\" (UniqueName: \"kubernetes.io/projected/cff7b24a-4b98-4592-9344-5ec4930c6079-kube-api-access-jbffg\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274312    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmv6\" (UniqueName: \"kubernetes.io/projected/528f5273-2e64-4225-b59b-1616674a15fa-kube-api-access-xrmv6\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274332    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274356    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274377    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274427    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-slash\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274455    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-node-log\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274533    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274550    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-lib-modules\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274595    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274671    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-tuned\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274659    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274716    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-out\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274766    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-tuned\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274800    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-bin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274806    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-out\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274857    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c927e4-b288-490a-acb9-c8e61846da1e-data-volume\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274889    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99d52cb6-4109-40d5-8fc7-fb6679a03d60-serviceca\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274917    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdgft\" (UniqueName: \"kubernetes.io/projected/25220e90-f330-4bd6-b2ff-45eb20b344c2-kube-api-access-wdgft\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274954    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c927e4-b288-490a-acb9-c8e61846da1e-data-volume\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274963    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-log-socket\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.274998    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275057    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-root\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275081    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-metrics-client-ca\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275099    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275128    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/380def42-3dee-46c1-83b8-52910cc74811-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275166    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275241    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55837b60-e43f-4eb2-9505-c755bde739ec-cert\") pod \"ingress-canary-n2d6j\" (UID: \"55837b60-e43f-4eb2-9505-c755bde739ec\") " pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275268    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275295    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-metrics-client-ca\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275313    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/380def42-3dee-46c1-83b8-52910cc74811-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275347    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-config-out\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275400    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275426    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-config-out\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275465    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c640c634-2704-48e7-a121-e0975ae43e23-iptables-alerter-script\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275483    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55837b60-e43f-4eb2-9505-c755bde739ec-cert\") pod \"ingress-canary-n2d6j\" (UID: \"55837b60-e43f-4eb2-9505-c755bde739ec\") " pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275523    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-sys\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275575    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-run\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275603    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-tmp-dir\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275606    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/99d52cb6-4109-40d5-8fc7-fb6679a03d60-serviceca\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275614    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275644    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275670    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-tmp-dir\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275680    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-config\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.275682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275709    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275797    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275806    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c640c634-2704-48e7-a121-e0975ae43e23-iptables-alerter-script\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275832    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cnibin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275860    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275869    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-config\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275929    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6b5\" (UniqueName: \"kubernetes.io/projected/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-kube-api-access-6q6b5\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.275974    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-var-lib-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276008    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qcz\" (UniqueName: \"kubernetes.io/projected/c640c634-2704-48e7-a121-e0975ae43e23-kube-api-access-v9qcz\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276016    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276030    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276057    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72ca4634-5648-402b-aaef-13f8d4ee2b4b-konnectivity-ca\") pod \"konnectivity-agent-pd8nn\" (UID: \"72ca4634-5648-402b-aaef-13f8d4ee2b4b\") " pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276123    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-systemd\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276163    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-system-cni-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276191    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-kubelet-config\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276261    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276282    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72ca4634-5648-402b-aaef-13f8d4ee2b4b-konnectivity-ca\") pod \"konnectivity-agent-pd8nn\" (UID: \"72ca4634-5648-402b-aaef-13f8d4ee2b4b\") " pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276298    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276410    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0175cfb-9276-42ed-bff8-f359a5727eb7-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276413    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-config\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276444    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5rq\" (UniqueName: \"kubernetes.io/projected/8e76a365-0c53-4369-b49e-b19a6611a391-kube-api-access-bx5rq\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276476    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5x76\" (UniqueName: \"kubernetes.io/projected/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-api-access-c5x76\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276505    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-client-certs\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276535    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-metrics-server-audit-profiles\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276567    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-netd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276594    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-tls\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276630    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72ca4634-5648-402b-aaef-13f8d4ee2b4b-agent-certs\") pod \"konnectivity-agent-pd8nn\" (UID: \"72ca4634-5648-402b-aaef-13f8d4ee2b4b\") " pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276647    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/528f5273-2e64-4225-b59b-1616674a15fa-ovnkube-config\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276686    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276721    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276748    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276749    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrckt\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-kube-api-access-vrckt\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276790    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7dc2b96a-b979-4de1-a69b-ce1c566518c6-secret-metrics-server-client-certs\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276801    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-k8s-cni-cncf-io\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276836    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-metrics-server-audit-profiles\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276854    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-tls\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276838    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-sys\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276908    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276923    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276953    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/528f5273-2e64-4225-b59b-1616674a15fa-ovn-node-metrics-cert\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276981    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.276981    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e76a365-0c53-4369-b49e-b19a6611a391-tmp-dir\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.277031    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-hostroot\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.277042    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e76a365-0c53-4369-b49e-b19a6611a391-tmp-dir\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.277073    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-etc-kubernetes\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.277110    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.277139    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-systemd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.277175    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/528f5273-2e64-4225-b59b-1616674a15fa-ovn-node-metrics-cert\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.277278 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.277279    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc2b96a-b979-4de1-a69b-ce1c566518c6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.278787 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.277340    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72ca4634-5648-402b-aaef-13f8d4ee2b4b-agent-certs\") pod \"konnectivity-agent-pd8nn\" (UID: \"72ca4634-5648-402b-aaef-13f8d4ee2b4b\") " pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:44:29.290306 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.290272    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9jp\" (UniqueName: \"kubernetes.io/projected/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-kube-api-access-cs9jp\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:44:29.290448 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.290405    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56w8p\" (UniqueName: \"kubernetes.io/projected/b0175cfb-9276-42ed-bff8-f359a5727eb7-kube-api-access-56w8p\") pod \"kube-state-metrics-5dc8d7d89b-k6t4c\" (UID: \"b0175cfb-9276-42ed-bff8-f359a5727eb7\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6t4c"
Nov 02 13:44:29.290831 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.290813    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkg7\" (UniqueName: \"kubernetes.io/projected/55837b60-e43f-4eb2-9505-c755bde739ec-kube-api-access-fjkg7\") pod \"ingress-canary-n2d6j\" (UID: \"55837b60-e43f-4eb2-9505-c755bde739ec\") " pod="openshift-ingress-canary/ingress-canary-n2d6j"
Nov 02 13:44:29.291991 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.291966    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtmnw\" (UniqueName: \"kubernetes.io/projected/99d52cb6-4109-40d5-8fc7-fb6679a03d60-kube-api-access-mtmnw\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:44:29.292139 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.292037    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsdb\" (UniqueName: \"kubernetes.io/projected/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-kube-api-access-vdsdb\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.292359 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.292337    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9222\" (UniqueName: \"kubernetes.io/projected/7dc2b96a-b979-4de1-a69b-ce1c566518c6-kube-api-access-q9222\") pod \"metrics-server-566f4c7487-mbcfx\" (UID: \"7dc2b96a-b979-4de1-a69b-ce1c566518c6\") " pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:29.292394 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.292339    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srg7z\" (UniqueName: \"kubernetes.io/projected/380def42-3dee-46c1-83b8-52910cc74811-kube-api-access-srg7z\") pod \"openshift-state-metrics-7bd7489957-dphbb\" (UID: \"380def42-3dee-46c1-83b8-52910cc74811\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dphbb"
Nov 02 13:44:29.293213 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.293193    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfr8\" (UniqueName: \"kubernetes.io/projected/094d4ae0-f6f8-44ae-a658-03f6322a3223-kube-api-access-8nfr8\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.293734 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.293715    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cczzh\" (UniqueName: \"kubernetes.io/projected/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-kube-api-access-cczzh\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.293947 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.293916    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdgft\" (UniqueName: \"kubernetes.io/projected/25220e90-f330-4bd6-b2ff-45eb20b344c2-kube-api-access-wdgft\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.293985 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.293947    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbffg\" (UniqueName: \"kubernetes.io/projected/cff7b24a-4b98-4592-9344-5ec4930c6079-kube-api-access-jbffg\") pod \"network-metrics-daemon-jcqf2\" (UID: \"cff7b24a-4b98-4592-9344-5ec4930c6079\") " pod="openshift-multus/network-metrics-daemon-jcqf2"
Nov 02 13:44:29.294192 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.294177    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjndc\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-kube-api-access-bjndc\") pod \"prometheus-k8s-0\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:29.295291 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.295275    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5rq\" (UniqueName: \"kubernetes.io/projected/8e76a365-0c53-4369-b49e-b19a6611a391-kube-api-access-bx5rq\") pod \"dns-default-lq2nc\" (UID: \"8e76a365-0c53-4369-b49e-b19a6611a391\") " pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:44:29.295556 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.295537    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6b5\" (UniqueName: \"kubernetes.io/projected/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-kube-api-access-6q6b5\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.295800 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.295785    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qcz\" (UniqueName: \"kubernetes.io/projected/c640c634-2704-48e7-a121-e0975ae43e23-kube-api-access-v9qcz\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:44:29.295884 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.295869    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5x76\" (UniqueName: \"kubernetes.io/projected/e6c927e4-b288-490a-acb9-c8e61846da1e-kube-api-access-c5x76\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.296921 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.296905    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7dkm\" (UniqueName: \"kubernetes.io/projected/9df3afdd-f643-4485-a85d-f78211c7b169-kube-api-access-g7dkm\") pod \"network-check-target-57d72\" (UID: \"9df3afdd-f643-4485-a85d-f78211c7b169\") " pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:44:29.302193 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.302164    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmv6\" (UniqueName: \"kubernetes.io/projected/528f5273-2e64-4225-b59b-1616674a15fa-kube-api-access-xrmv6\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.302369 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.302345    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrckt\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-kube-api-access-vrckt\") pod \"alertmanager-main-0\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:29.377853 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377761    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-netns\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.377853 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377799    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-socket-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.377853 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377815    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e6c927e4-b288-490a-acb9-c8e61846da1e-crio-socket\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.377853 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377850    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d52cb6-4109-40d5-8fc7-fb6679a03d60-host\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377877    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-ovn\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377894    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-netns\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377914    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-socket-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377931    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99d52cb6-4109-40d5-8fc7-fb6679a03d60-host\") pod \"node-ca-gkg2v\" (UID: \"99d52cb6-4109-40d5-8fc7-fb6679a03d60\") " pod="openshift-image-registry/node-ca-gkg2v"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377931    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e6c927e4-b288-490a-acb9-c8e61846da1e-crio-socket\") pod \"insights-runtime-extractor-lrzvt\" (UID: \"e6c927e4-b288-490a-acb9-c8e61846da1e\") " pod="openshift-insights/insights-runtime-extractor-lrzvt"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377971    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-multus-certs\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377902    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-multus-certs\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378022    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cnibin\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.377971    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-ovn\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378051    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378076    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-modprobe-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378070    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-cnibin\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378110    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378131    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378146    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-kubelet\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378180    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-device-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378196    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378205    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-dbus\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378210    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-modprobe-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378220    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-kubelet\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378238    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-wtmp\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.378240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378254    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-dbus\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378247    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-device-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378275    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-hosts-file\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378311    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-slash\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378335    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-node-log\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378371    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-lib-modules\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378396    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35941a8e-d549-436f-b3e1-0be3ef2cc3a7-hosts-file\") pod \"node-resolver-5jbwx\" (UID: \"35941a8e-d549-436f-b3e1-0be3ef2cc3a7\") " pod="openshift-dns/node-resolver-5jbwx"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378401    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-node-log\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378401    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-bin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378397    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-slash\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378446    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-log-socket\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378450    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-bin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378492    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-root\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378515    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378546    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-lib-modules\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378550    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-log-socket\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378564    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378570    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378578    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-root\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378589    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-sys\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378609    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-run\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378627    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378643    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cnibin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378652    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-d\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378659    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-var-lib-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378665    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-sys\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378670    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-run\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378711    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-var-lib-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378721    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378741    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-cnibin\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378748    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378381    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fad8eb6b-fa80-41f6-b5ff-51c39bba446c-node-exporter-wtmp\") pod \"node-exporter-jgpx4\" (UID: \"fad8eb6b-fa80-41f6-b5ff-51c39bba446c\") " pod="openshift-monitoring/node-exporter-jgpx4"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378828    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-systemd\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378874    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378889    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-system-cni-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378910    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-systemd\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378917    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-kubelet-config\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378924    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac48fd07-ce48-4fc0-afa0-e9b31baaaec4-system-cni-dir\") pod \"multus-additional-cni-plugins-cgnfh\" (UID: \"ac48fd07-ce48-4fc0-afa0-e9b31baaaec4\") " pod="openshift-multus/multus-additional-cni-plugins-cgnfh"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378966    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.378974    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/570c7233-f4c2-4c16-b9d9-67baa3fa7b96-kubelet-config\") pod \"global-pull-secret-syncer-vsx66\" (UID: \"570c7233-f4c2-4c16-b9d9-67baa3fa7b96\") " pod="kube-system/global-pull-secret-syncer-vsx66"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379008    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379030    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-netd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379055    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-k8s-cni-cncf-io\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379073    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-sys\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379093    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-hostroot\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.379076 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379109    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-etc-kubernetes\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379125    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-systemd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379131    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-run-k8s-cni-cncf-io\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379146    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-sys\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379150    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-hostroot\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379162    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-etc-kubernetes\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379150    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-conf\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379181    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-systemd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379184    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-netd\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379204    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-host\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379224    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-conf-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379258    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-conf-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379259    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-host\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379281    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-sys-fs\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379299    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-kubelet\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379323    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysconfig\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379321    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-sys-fs\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379341    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-kubernetes\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379350    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysctl-conf\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379361    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-multus\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379362    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-kubelet\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379381    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-sysconfig\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379387    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-host-var-lib-cni-multus\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379407    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-etc-kubernetes\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379405    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-registration-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379431    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-systemd-units\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379450    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/094d4ae0-f6f8-44ae-a658-03f6322a3223-registration-dir\") pod \"aws-ebs-csi-driver-node-wn29k\" (UID: \"094d4ae0-f6f8-44ae-a658-03f6322a3223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn29k"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379466    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379498    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-netns\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379529    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-var-lib-kubelet\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379532    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-systemd-units\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379548    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-run-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379551    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-socket-dir-parent\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379587    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-run-netns\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379597    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-multus-socket-dir-parent\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379604    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-etc-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379612    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25220e90-f330-4bd6-b2ff-45eb20b344c2-var-lib-kubelet\") pod \"tuned-xgnrc\" (UID: \"25220e90-f330-4bd6-b2ff-45eb20b344c2\") " pod="openshift-cluster-node-tuning-operator/tuned-xgnrc"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379632    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-bin\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379647    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-etc-openvswitch\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379665    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/528f5273-2e64-4225-b59b-1616674a15fa-host-cni-bin\") pod \"ovnkube-node-sr6pw\" (UID: \"528f5273-2e64-4225-b59b-1616674a15fa\") " pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379688    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-system-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379721    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c640c634-2704-48e7-a121-e0975ae43e23-host-slash\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379765    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb-system-cni-dir\") pod \"multus-zf8tf\" (UID: \"2f6bae19-4b38-4f5e-8ce6-bc3ac51eecbb\") " pod="openshift-multus/multus-zf8tf"
Nov 02 13:44:29.380506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.379814    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c640c634-2704-48e7-a121-e0975ae43e23-host-slash\") pod \"iptables-alerter-4j587\" (UID: \"c640c634-2704-48e7-a121-e0975ae43e23\") " pod="openshift-network-operator/iptables-alerter-4j587"
Nov 02 13:44:29.733273 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:29.733231    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:30.137521 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:30.137425    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:44:31.935555 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:31.935516    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lq2nc"
Nov 02 13:44:32.221874 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:32.221777    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:34.552698 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:34.552672    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:34.553111 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:34.552906    6284 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:44:38.906316 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:38.906279    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sr6pw"
Nov 02 13:44:43.141201 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:43.141164    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-566f4c7487-mbcfx"
Nov 02 13:44:43.896162 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:43.896116    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:44:44.099852 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:44.099821    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pd8nn"
Nov 02 13:44:48.909387 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:48.909351    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:48.909911 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:48.909740    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="alertmanager" containerID="cri-o://0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4" gracePeriod=120
Nov 02 13:44:48.910039 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:48.910003    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy" containerID="cri-o://a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4" gracePeriod=120
Nov 02 13:44:48.910121 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:48.910048    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="config-reloader" containerID="cri-o://6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19" gracePeriod=120
Nov 02 13:44:48.910121 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:48.910045    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="prom-label-proxy" containerID="cri-o://5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83" gracePeriod=120
Nov 02 13:44:48.910284 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:48.910001    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy-web" containerID="cri-o://d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b" gracePeriod=120
Nov 02 13:44:48.910284 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:48.910099    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy-metric" containerID="cri-o://b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e" gracePeriod=120
Nov 02 13:44:49.319277 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:49.319242    6284 generic.go:356] "Generic (PLEG): container finished" podID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerID="5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83" exitCode=0
Nov 02 13:44:49.319277 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:49.319272    6284 generic.go:356] "Generic (PLEG): container finished" podID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerID="b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e" exitCode=0
Nov 02 13:44:49.319277 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:49.319281    6284 generic.go:356] "Generic (PLEG): container finished" podID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerID="a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4" exitCode=0
Nov 02 13:44:49.319277 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:49.319291    6284 generic.go:356] "Generic (PLEG): container finished" podID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerID="6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19" exitCode=0
Nov 02 13:44:49.319277 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:49.319299    6284 generic.go:356] "Generic (PLEG): container finished" podID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerID="0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4" exitCode=0
Nov 02 13:44:49.319605 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:49.319321    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerDied","Data":"5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83"}
Nov 02 13:44:49.319605 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:49.319414    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerDied","Data":"b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e"}
Nov 02 13:44:49.319605 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:49.319442    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerDied","Data":"a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4"}
Nov 02 13:44:49.319605 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:49.319467    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerDied","Data":"6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19"}
Nov 02 13:44:49.319605 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:49.319491    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerDied","Data":"0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4"}
Nov 02 13:44:50.164204 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.164173    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.261858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.261759    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-trusted-ca-bundle\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.261858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.261800    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-volume\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-volume\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.261858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.261831    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.261858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.261859    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-tls-assets\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-tls-assets\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.262254 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.262005    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-metrics-client-ca\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.262254 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.262052    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-out\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.262254 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.262079    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-cluster-tls-config\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.262254 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.262125    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-main-tls\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.262254 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.262158    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-kube-api-access-vrckt\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-kube-api-access-vrckt\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.262254 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.262189    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.262254 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.262228    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-main-db\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.262674 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.262261    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-web-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-web-config\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.262674 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.262302    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-web\") pod \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\" (UID: \"947938ad-88e5-4ebd-aefe-94ca1b32e7f7\") "
Nov 02 13:44:50.262789 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.262616    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-trusted-ca-bundle" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:50.263269 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.263083    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-metrics-client-ca" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:50.263398 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.263298    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-main-db" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:50.266176 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.266004    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-web" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:50.266420 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.266370    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-main-tls" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:50.267306 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.267278    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-metric" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:50.267622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.267601    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-tls-assets" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:50.267721 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.267646    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-volume" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:50.268437 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.268404    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:50.268685 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.268662    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-kube-api-access-vrckt" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "kube-api-access-vrckt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:50.271778 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.271732    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-out" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:50.272671 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.272646    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-cluster-tls-config" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:50.279569 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.279524    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-web-config" pod "947938ad-88e5-4ebd-aefe-94ca1b32e7f7" (UID: "947938ad-88e5-4ebd-aefe-94ca1b32e7f7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:50.325854 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.325820    6284 generic.go:356] "Generic (PLEG): container finished" podID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerID="d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b" exitCode=0
Nov 02 13:44:50.326052 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.325918    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.326052 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.325907    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerDied","Data":"d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b"}
Nov 02 13:44:50.326148 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.326087    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"947938ad-88e5-4ebd-aefe-94ca1b32e7f7","Type":"ContainerDied","Data":"9b2ed9769382472c01d45016c359d28f9c73ca0b93ec11ff8681d526316a90f2"}
Nov 02 13:44:50.326148 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.326140    6284 scope.go:122] "RemoveContainer" containerID="5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83"
Nov 02 13:44:50.334189 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.334164    6284 scope.go:122] "RemoveContainer" containerID="b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e"
Nov 02 13:44:50.341858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.341836    6284 scope.go:122] "RemoveContainer" containerID="a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4"
Nov 02 13:44:50.349643 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.349623    6284 scope.go:122] "RemoveContainer" containerID="d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b"
Nov 02 13:44:50.357414 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.357392    6284 scope.go:122] "RemoveContainer" containerID="6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19"
Nov 02 13:44:50.363827 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363803    6284 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-cluster-tls-config\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.363827 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363831    6284 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-main-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363845    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrckt\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-kube-api-access-vrckt\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363857    6284 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363866    6284 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-main-db\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363901    6284 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-web-config\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363912    6284 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363922    6284 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363943    6284 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363956    6284 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363965    6284 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-tls-assets\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363974    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-metrics-client-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.364011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.363984    6284 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/947938ad-88e5-4ebd-aefe-94ca1b32e7f7-config-out\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:50.365393 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.365379    6284 scope.go:122] "RemoveContainer" containerID="0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4"
Nov 02 13:44:50.373512 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.373490    6284 scope.go:122] "RemoveContainer" containerID="e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741"
Nov 02 13:44:50.375128 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.374972    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:50.379807 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.379774    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:50.382714 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.382696    6284 scope.go:122] "RemoveContainer" containerID="5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83"
Nov 02 13:44:50.383065 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:50.383044    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83\": container with ID starting with 5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83 not found: ID does not exist" containerID="5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83"
Nov 02 13:44:50.383121 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.383076    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83"} err="failed to get container status \"5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83\": rpc error: code = NotFound desc = could not find container \"5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83\": container with ID starting with 5d7cdf62dd62120ad49d1e7e37b99ba400d9f773f7470b514f4c698dfe4cba83 not found: ID does not exist"
Nov 02 13:44:50.383121 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.383114    6284 scope.go:122] "RemoveContainer" containerID="b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e"
Nov 02 13:44:50.383434 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:50.383412    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e\": container with ID starting with b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e not found: ID does not exist" containerID="b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e"
Nov 02 13:44:50.383475 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.383443    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e"} err="failed to get container status \"b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e\": rpc error: code = NotFound desc = could not find container \"b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e\": container with ID starting with b9aecf2655e0d5543884ef0e08cf608aaeee354a8403d1dbc4b6f8ad3a37566e not found: ID does not exist"
Nov 02 13:44:50.383475 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.383464    6284 scope.go:122] "RemoveContainer" containerID="a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4"
Nov 02 13:44:50.383729 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:50.383712    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4\": container with ID starting with a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4 not found: ID does not exist" containerID="a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4"
Nov 02 13:44:50.383768 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.383735    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4"} err="failed to get container status \"a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4\": rpc error: code = NotFound desc = could not find container \"a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4\": container with ID starting with a8cfe70fbdf5514261621ec151d0cb2e2ca22e0433692a60f5f3205edf38b4c4 not found: ID does not exist"
Nov 02 13:44:50.383768 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.383751    6284 scope.go:122] "RemoveContainer" containerID="d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b"
Nov 02 13:44:50.384039 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:50.384022    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b\": container with ID starting with d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b not found: ID does not exist" containerID="d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b"
Nov 02 13:44:50.384104 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.384043    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b"} err="failed to get container status \"d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b\": rpc error: code = NotFound desc = could not find container \"d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b\": container with ID starting with d2ba5d07a397b2bbb7f56fdff06bae3d9595615210b7dcf312189d6265db499b not found: ID does not exist"
Nov 02 13:44:50.384104 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.384058    6284 scope.go:122] "RemoveContainer" containerID="6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19"
Nov 02 13:44:50.384330 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:50.384314    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19\": container with ID starting with 6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19 not found: ID does not exist" containerID="6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19"
Nov 02 13:44:50.384406 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.384333    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19"} err="failed to get container status \"6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19\": rpc error: code = NotFound desc = could not find container \"6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19\": container with ID starting with 6321c49ae5bdf427ee85c167158e0aad601d99a28a0535aa75b7f1aec96fdb19 not found: ID does not exist"
Nov 02 13:44:50.384406 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.384348    6284 scope.go:122] "RemoveContainer" containerID="0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4"
Nov 02 13:44:50.384612 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:50.384575    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4\": container with ID starting with 0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4 not found: ID does not exist" containerID="0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4"
Nov 02 13:44:50.384612 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.384601    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4"} err="failed to get container status \"0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4\": rpc error: code = NotFound desc = could not find container \"0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4\": container with ID starting with 0ef9550faf5d842140137a56beec10b207f704ecc4d3fe7cb9c088b4311b5ba4 not found: ID does not exist"
Nov 02 13:44:50.384746 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.384619    6284 scope.go:122] "RemoveContainer" containerID="e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741"
Nov 02 13:44:50.384966 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:50.384921    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741\": container with ID starting with e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741 not found: ID does not exist" containerID="e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741"
Nov 02 13:44:50.385069 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.384972    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741"} err="failed to get container status \"e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741\": rpc error: code = NotFound desc = could not find container \"e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741\": container with ID starting with e65c360d413ee951849e3634d4723c9e9f89169bbdf98165b9517341e14b7741 not found: ID does not exist"
Nov 02 13:44:50.412954 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.412899    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:50.413197 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413183    6284 cpu_manager.go:414] "containerMap: removing container" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="init-config-reloader"
Nov 02 13:44:50.413239 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413199    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="init-config-reloader"
Nov 02 13:44:50.413239 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413217    6284 cpu_manager.go:414] "containerMap: removing container" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy-web"
Nov 02 13:44:50.413239 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413222    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy-web"
Nov 02 13:44:50.413239 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413231    6284 cpu_manager.go:414] "containerMap: removing container" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy"
Nov 02 13:44:50.413239 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413235    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413249    6284 cpu_manager.go:414] "containerMap: removing container" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="prom-label-proxy"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413254    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="prom-label-proxy"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413260    6284 cpu_manager.go:414] "containerMap: removing container" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy-metric"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413264    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy-metric"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413274    6284 cpu_manager.go:414] "containerMap: removing container" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="config-reloader"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413279    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="config-reloader"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413290    6284 cpu_manager.go:414] "containerMap: removing container" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="alertmanager"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413294    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="alertmanager"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413333    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="prom-label-proxy"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413341    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy-metric"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413349    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="config-reloader"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413355    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="alertmanager"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413360    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="init-config-reloader"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413366    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy-web"
Nov 02 13:44:50.413412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.413371    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" containerName="kube-rbac-proxy"
Nov 02 13:44:50.417279 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.417256    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.430779 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.430737    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 13:44:50.430779 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.430761    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 13:44:50.430779 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.430767    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 13:44:50.430779 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.430754    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 13:44:50.431114 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.430795    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 13:44:50.431114 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.430800    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 13:44:50.431114 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.430779    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 13:44:50.431114 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.430856    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 13:44:50.431114 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.430984    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rxjnr\""
Nov 02 13:44:50.440353 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.440323    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 13:44:50.444588 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.444555    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:50.465092 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465056    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-config-out\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465092 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465095    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkjqw\" (UniqueName: \"kubernetes.io/projected/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-kube-api-access-wkjqw\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465355 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465113    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-config-volume\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465355 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465131    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465355 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465204    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465355 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465353    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465537 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465371    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465537 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465398    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465537 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465449    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465703 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465550    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465703 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465581    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465703 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465600    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.465703 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.465622    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-web-config\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.566357 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.566240    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-config-out\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.566357 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.566292    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkjqw\" (UniqueName: \"kubernetes.io/projected/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-kube-api-access-wkjqw\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.566357 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.566326    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-config-volume\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.566357 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.566353    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.566743 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.566410    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.566743 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.566467    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.566743 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.566497    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.566743 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.566642    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.566743 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.566677    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.567077 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.567052    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.567166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.567126    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.567166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.567123    6284 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 13:44:50.567277 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.567179    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.567277 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.567232    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-web-config\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.568012 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.567982    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.568012 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.568003    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.568685 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.568660    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.570374 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.570297    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-config-volume\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.570489 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.570422    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-web-config\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.574300 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.574186    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-config-out\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.574300 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.574274    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.574300 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.574287    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.574300 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.574304    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.574585 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.574279    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.574833 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.574812    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.575346 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.575328    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.588172 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.588146    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkjqw\" (UniqueName: \"kubernetes.io/projected/6f7554a7-ea8c-401f-853e-35b1dcc01d9f-kube-api-access-wkjqw\") pod \"alertmanager-main-0\" (UID: \"6f7554a7-ea8c-401f-853e-35b1dcc01d9f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:50.738524 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:50.738490    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 13:44:51.068319 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:51.068282    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 13:44:51.073489 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:51.073452    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7554a7_ea8c_401f_853e_35b1dcc01d9f.slice/crio-9cd175f74fa3cc438982f68da12c82c1db6b28c81b8d668f43b43d7a3134fd25 WatchSource:0}: Error finding container 9cd175f74fa3cc438982f68da12c82c1db6b28c81b8d668f43b43d7a3134fd25: Status 404 returned error can't find the container with id 9cd175f74fa3cc438982f68da12c82c1db6b28c81b8d668f43b43d7a3134fd25
Nov 02 13:44:51.332176 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:51.332079    6284 generic.go:356] "Generic (PLEG): container finished" podID="6f7554a7-ea8c-401f-853e-35b1dcc01d9f" containerID="0fc452fca9282edd12390e0eb47eed8ed638ce32389b4143a11d09a3e69bbfa3" exitCode=0
Nov 02 13:44:51.332557 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:51.332173    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7554a7-ea8c-401f-853e-35b1dcc01d9f","Type":"ContainerDied","Data":"0fc452fca9282edd12390e0eb47eed8ed638ce32389b4143a11d09a3e69bbfa3"}
Nov 02 13:44:51.332557 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:51.332273    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7554a7-ea8c-401f-853e-35b1dcc01d9f","Type":"ContainerStarted","Data":"9cd175f74fa3cc438982f68da12c82c1db6b28c81b8d668f43b43d7a3134fd25"}
Nov 02 13:44:52.167543 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:52.167508    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947938ad-88e5-4ebd-aefe-94ca1b32e7f7" path="/var/lib/kubelet/pods/947938ad-88e5-4ebd-aefe-94ca1b32e7f7/volumes"
Nov 02 13:44:52.339669 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:52.339583    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7554a7-ea8c-401f-853e-35b1dcc01d9f","Type":"ContainerStarted","Data":"e3eeaea3ce3f1701916bea807c280ba4c1a3d23d835b0d2fc458c6cc8086f28d"}
Nov 02 13:44:52.339669 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:52.339654    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7554a7-ea8c-401f-853e-35b1dcc01d9f","Type":"ContainerStarted","Data":"32b2bade6a86e1299dcbc7b74beab08b0e3140d92d21b6c493762835ebeef132"}
Nov 02 13:44:52.340103 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:52.339683    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7554a7-ea8c-401f-853e-35b1dcc01d9f","Type":"ContainerStarted","Data":"ab19517c0c53147032a954dde936d5ac63f1ce04283fa8d6e01eead542e56223"}
Nov 02 13:44:52.340103 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:52.339707    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7554a7-ea8c-401f-853e-35b1dcc01d9f","Type":"ContainerStarted","Data":"6af8642a66ac55ac959391ac0ff40a9c2d4b848e95069d539d1950da02fa773d"}
Nov 02 13:44:52.340103 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:52.339738    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7554a7-ea8c-401f-853e-35b1dcc01d9f","Type":"ContainerStarted","Data":"0efd52726d87911e564f3b234190ddda97640c2f04081331a5552f575f584b75"}
Nov 02 13:44:52.340103 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:52.339762    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7554a7-ea8c-401f-853e-35b1dcc01d9f","Type":"ContainerStarted","Data":"57d20f35be147c138ba9a3f839413443821552234cf83c14a4394c18a2d39e10"}
Nov 02 13:44:52.399096 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:52.399037    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.385786375 podStartE2EDuration="2.399022454s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.236079ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:52.398323569 +0000 UTC m=+24.357699178" watchObservedRunningTime="2026-11-02 13:44:52.399022454 +0000 UTC m=+24.358398058"
Nov 02 13:44:53.498524 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.498491    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 13:44:53.499007 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.498779    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="prometheus" containerID="cri-o://39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a" gracePeriod=600
Nov 02 13:44:53.499007 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.498800    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy" containerID="cri-o://9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591" gracePeriod=600
Nov 02 13:44:53.499007 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.498845    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy-thanos" containerID="cri-o://273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e" gracePeriod=600
Nov 02 13:44:53.499268 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.498826    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy-web" containerID="cri-o://3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d" gracePeriod=600
Nov 02 13:44:53.499411 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.499269    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="config-reloader" containerID="cri-o://6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748" gracePeriod=600
Nov 02 13:44:53.499411 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.499202    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="thanos-sidecar" containerID="cri-o://4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b" gracePeriod=600
Nov 02 13:44:53.762926 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.762896    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:53.793460 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793426    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.793627 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793468    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-kube-rbac-proxy\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.793627 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793510    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-trusted-ca-bundle\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.793627 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793538    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-config-out\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-config-out\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.793791 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793625    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-serving-certs-ca-bundle\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.793791 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793656    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-1\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.793791 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793683    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-grpc-tls\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.793791 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793715    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-2\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.793791 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793744    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-web-config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-web-config\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.793791 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793788    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-db\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.794112 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793823    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-thanos-prometheus-http-client-file\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.794112 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793854    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-config\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.794112 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793920    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-kubelet-serving-ca-bundle\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.794112 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.793980    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-kube-api-access-bjndc\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-kube-api-access-bjndc\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.794112 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.794010    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-tls-assets\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-tls-assets\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.794112 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.794019    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-trusted-ca-bundle" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:53.794611 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.794579    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-kubelet-serving-ca-bundle" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:53.794695 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.794677    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-serving-certs-ca-bundle" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:53.795864 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.794778    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-2" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:53.795864 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.794856    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-1" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:53.796616 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796146    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.796616 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796210    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-db" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:53.796616 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796217    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-metrics-client-ca\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.796616 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796284    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-metrics-client-certs\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.796616 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796325    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-0\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.796616 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796368    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-tls\") pod \"da5bed06-f76d-474a-88ed-1fc6be553533\" (UID: \"da5bed06-f76d-474a-88ed-1fc6be553533\") "
Nov 02 13:44:53.797331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796787    6284 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.797331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796864    6284 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-db\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.797331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796889    6284 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.797331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796900    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-metrics-client-ca" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:53.797331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796907    6284 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.797331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796974    6284 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.797331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.796994    6284 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.797331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.797124    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-config" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:53.797331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.797209    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-thanos-sidecar-tls" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:53.797331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.797276    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-config-out" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:53.797807 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.797532    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-tls-assets" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:53.797807 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.797531    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-kube-api-access-bjndc" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "kube-api-access-bjndc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:53.797927 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.797899    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-thanos-prometheus-http-client-file" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:53.798186 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.798157    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-kube-rbac-proxy" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:53.798298 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.798274    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-grpc-tls" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:53.798417 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.798392    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-0" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:53.799141 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.799114    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-kube-rbac-proxy-web" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:53.799251 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.799170    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-metrics-client-certs" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:53.799582 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.799567    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-tls" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:53.808773 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.808738    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-web-config" pod "da5bed06-f76d-474a-88ed-1fc6be553533" (UID: "da5bed06-f76d-474a-88ed-1fc6be553533"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:53.898020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.897973    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjndc\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-kube-api-access-bjndc\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898010    6284 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da5bed06-f76d-474a-88ed-1fc6be553533-tls-assets\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898023    6284 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898032    6284 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-configmap-metrics-client-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898042    6284 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-metrics-client-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898051    6284 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da5bed06-f76d-474a-88ed-1fc6be553533-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898059    6284 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898068    6284 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898077    6284 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-kube-rbac-proxy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898086    6284 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da5bed06-f76d-474a-88ed-1fc6be553533-config-out\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898095    6284 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-secret-grpc-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898125    6284 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-web-config\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898134    6284 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:53.898289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:53.898142    6284 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/da5bed06-f76d-474a-88ed-1fc6be553533-config\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:44:54.349332 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349292    6284 generic.go:356] "Generic (PLEG): container finished" podID="da5bed06-f76d-474a-88ed-1fc6be553533" containerID="273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e" exitCode=0
Nov 02 13:44:54.349332 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349316    6284 generic.go:356] "Generic (PLEG): container finished" podID="da5bed06-f76d-474a-88ed-1fc6be553533" containerID="9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591" exitCode=0
Nov 02 13:44:54.349332 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349324    6284 generic.go:356] "Generic (PLEG): container finished" podID="da5bed06-f76d-474a-88ed-1fc6be553533" containerID="3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d" exitCode=0
Nov 02 13:44:54.349332 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349333    6284 generic.go:356] "Generic (PLEG): container finished" podID="da5bed06-f76d-474a-88ed-1fc6be553533" containerID="4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b" exitCode=0
Nov 02 13:44:54.349332 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349340    6284 generic.go:356] "Generic (PLEG): container finished" podID="da5bed06-f76d-474a-88ed-1fc6be553533" containerID="6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748" exitCode=0
Nov 02 13:44:54.349332 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349346    6284 generic.go:356] "Generic (PLEG): container finished" podID="da5bed06-f76d-474a-88ed-1fc6be553533" containerID="39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a" exitCode=0
Nov 02 13:44:54.349846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349385    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.349846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349384    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerDied","Data":"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"}
Nov 02 13:44:54.349846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349475    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerDied","Data":"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"}
Nov 02 13:44:54.349846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349519    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerDied","Data":"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"}
Nov 02 13:44:54.349846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349541    6284 scope.go:122] "RemoveContainer" containerID="273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"
Nov 02 13:44:54.349846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349555    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerDied","Data":"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"}
Nov 02 13:44:54.349846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349597    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerDied","Data":"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"}
Nov 02 13:44:54.349846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349638    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerDied","Data":"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"}
Nov 02 13:44:54.349846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.349678    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da5bed06-f76d-474a-88ed-1fc6be553533","Type":"ContainerDied","Data":"293dddafbc0ac9b18beefb2d7c6b1ae612e0a25d3f68aaa89a0f5671c5ca083b"}
Nov 02 13:44:54.357505 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.357482    6284 scope.go:122] "RemoveContainer" containerID="9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"
Nov 02 13:44:54.365764 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.365739    6284 scope.go:122] "RemoveContainer" containerID="3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"
Nov 02 13:44:54.373504 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.373476    6284 scope.go:122] "RemoveContainer" containerID="4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"
Nov 02 13:44:54.381563 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.381543    6284 scope.go:122] "RemoveContainer" containerID="6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"
Nov 02 13:44:54.390342 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.390318    6284 scope.go:122] "RemoveContainer" containerID="39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"
Nov 02 13:44:54.398565 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.398546    6284 scope.go:122] "RemoveContainer" containerID="e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"
Nov 02 13:44:54.406079 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.406054    6284 scope.go:122] "RemoveContainer" containerID="273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"
Nov 02 13:44:54.406438 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:54.406421    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": container with ID starting with 273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e not found: ID does not exist" containerID="273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"
Nov 02 13:44:54.406483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.406453    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"} err="failed to get container status \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": rpc error: code = NotFound desc = could not find container \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": container with ID starting with 273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e not found: ID does not exist"
Nov 02 13:44:54.406483 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.406472    6284 scope.go:122] "RemoveContainer" containerID="9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"
Nov 02 13:44:54.406752 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:54.406734    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": container with ID starting with 9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591 not found: ID does not exist" containerID="9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"
Nov 02 13:44:54.406811 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.406756    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"} err="failed to get container status \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": rpc error: code = NotFound desc = could not find container \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": container with ID starting with 9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591 not found: ID does not exist"
Nov 02 13:44:54.406811 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.406771    6284 scope.go:122] "RemoveContainer" containerID="3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"
Nov 02 13:44:54.407116 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:54.407094    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": container with ID starting with 3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d not found: ID does not exist" containerID="3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"
Nov 02 13:44:54.407194 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.407122    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"} err="failed to get container status \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": rpc error: code = NotFound desc = could not find container \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": container with ID starting with 3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d not found: ID does not exist"
Nov 02 13:44:54.407194 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.407139    6284 scope.go:122] "RemoveContainer" containerID="4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"
Nov 02 13:44:54.407400 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:54.407383    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": container with ID starting with 4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b not found: ID does not exist" containerID="4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"
Nov 02 13:44:54.407460 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.407403    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"} err="failed to get container status \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": rpc error: code = NotFound desc = could not find container \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": container with ID starting with 4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b not found: ID does not exist"
Nov 02 13:44:54.407460 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.407417    6284 scope.go:122] "RemoveContainer" containerID="6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"
Nov 02 13:44:54.407676 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:54.407657    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": container with ID starting with 6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748 not found: ID does not exist" containerID="6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"
Nov 02 13:44:54.407728 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.407686    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"} err="failed to get container status \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": rpc error: code = NotFound desc = could not find container \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": container with ID starting with 6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748 not found: ID does not exist"
Nov 02 13:44:54.407728 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.407707    6284 scope.go:122] "RemoveContainer" containerID="39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"
Nov 02 13:44:54.407955 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:54.407919    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": container with ID starting with 39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a not found: ID does not exist" containerID="39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"
Nov 02 13:44:54.408017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.407953    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"} err="failed to get container status \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": rpc error: code = NotFound desc = could not find container \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": container with ID starting with 39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a not found: ID does not exist"
Nov 02 13:44:54.408017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.407971    6284 scope.go:122] "RemoveContainer" containerID="e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"
Nov 02 13:44:54.408235 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:44:54.408219    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": container with ID starting with e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504 not found: ID does not exist" containerID="e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"
Nov 02 13:44:54.408290 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.408239    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"} err="failed to get container status \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": rpc error: code = NotFound desc = could not find container \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": container with ID starting with e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504 not found: ID does not exist"
Nov 02 13:44:54.408290 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.408251    6284 scope.go:122] "RemoveContainer" containerID="273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"
Nov 02 13:44:54.408485 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.408465    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 13:44:54.408485 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.408482    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"} err="failed to get container status \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": rpc error: code = NotFound desc = could not find container \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": container with ID starting with 273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e not found: ID does not exist"
Nov 02 13:44:54.408485 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.408498    6284 scope.go:122] "RemoveContainer" containerID="9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"
Nov 02 13:44:54.408750 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.408727    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"} err="failed to get container status \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": rpc error: code = NotFound desc = could not find container \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": container with ID starting with 9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591 not found: ID does not exist"
Nov 02 13:44:54.408809 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.408750    6284 scope.go:122] "RemoveContainer" containerID="3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"
Nov 02 13:44:54.409104 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.409082    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"} err="failed to get container status \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": rpc error: code = NotFound desc = could not find container \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": container with ID starting with 3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d not found: ID does not exist"
Nov 02 13:44:54.409104 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.409101    6284 scope.go:122] "RemoveContainer" containerID="4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"
Nov 02 13:44:54.409367 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.409350    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"} err="failed to get container status \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": rpc error: code = NotFound desc = could not find container \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": container with ID starting with 4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b not found: ID does not exist"
Nov 02 13:44:54.409423 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.409367    6284 scope.go:122] "RemoveContainer" containerID="6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"
Nov 02 13:44:54.409586 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.409566    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"} err="failed to get container status \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": rpc error: code = NotFound desc = could not find container \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": container with ID starting with 6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748 not found: ID does not exist"
Nov 02 13:44:54.409629 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.409587    6284 scope.go:122] "RemoveContainer" containerID="39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"
Nov 02 13:44:54.409803 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.409787    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"} err="failed to get container status \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": rpc error: code = NotFound desc = could not find container \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": container with ID starting with 39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a not found: ID does not exist"
Nov 02 13:44:54.409843 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.409803    6284 scope.go:122] "RemoveContainer" containerID="e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"
Nov 02 13:44:54.410052 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.410031    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"} err="failed to get container status \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": rpc error: code = NotFound desc = could not find container \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": container with ID starting with e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504 not found: ID does not exist"
Nov 02 13:44:54.410105 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.410052    6284 scope.go:122] "RemoveContainer" containerID="273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"
Nov 02 13:44:54.410310 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.410290    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"} err="failed to get container status \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": rpc error: code = NotFound desc = could not find container \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": container with ID starting with 273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e not found: ID does not exist"
Nov 02 13:44:54.410352 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.410310    6284 scope.go:122] "RemoveContainer" containerID="9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"
Nov 02 13:44:54.410538 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.410518    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"} err="failed to get container status \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": rpc error: code = NotFound desc = could not find container \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": container with ID starting with 9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591 not found: ID does not exist"
Nov 02 13:44:54.410578 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.410538    6284 scope.go:122] "RemoveContainer" containerID="3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"
Nov 02 13:44:54.410746 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.410727    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"} err="failed to get container status \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": rpc error: code = NotFound desc = could not find container \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": container with ID starting with 3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d not found: ID does not exist"
Nov 02 13:44:54.410787 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.410746    6284 scope.go:122] "RemoveContainer" containerID="4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"
Nov 02 13:44:54.410968 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.410945    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"} err="failed to get container status \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": rpc error: code = NotFound desc = could not find container \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": container with ID starting with 4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b not found: ID does not exist"
Nov 02 13:44:54.410968 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.410964    6284 scope.go:122] "RemoveContainer" containerID="6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"
Nov 02 13:44:54.411210 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.411192    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"} err="failed to get container status \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": rpc error: code = NotFound desc = could not find container \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": container with ID starting with 6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748 not found: ID does not exist"
Nov 02 13:44:54.411259 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.411210    6284 scope.go:122] "RemoveContainer" containerID="39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"
Nov 02 13:44:54.411434 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.411416    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"} err="failed to get container status \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": rpc error: code = NotFound desc = could not find container \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": container with ID starting with 39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a not found: ID does not exist"
Nov 02 13:44:54.411477 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.411435    6284 scope.go:122] "RemoveContainer" containerID="e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"
Nov 02 13:44:54.411690 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.411673    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"} err="failed to get container status \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": rpc error: code = NotFound desc = could not find container \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": container with ID starting with e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504 not found: ID does not exist"
Nov 02 13:44:54.411732 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.411691    6284 scope.go:122] "RemoveContainer" containerID="273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"
Nov 02 13:44:54.411929 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.411909    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"} err="failed to get container status \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": rpc error: code = NotFound desc = could not find container \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": container with ID starting with 273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e not found: ID does not exist"
Nov 02 13:44:54.412016 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.411930    6284 scope.go:122] "RemoveContainer" containerID="9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"
Nov 02 13:44:54.412213 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.412194    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"} err="failed to get container status \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": rpc error: code = NotFound desc = could not find container \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": container with ID starting with 9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591 not found: ID does not exist"
Nov 02 13:44:54.412260 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.412213    6284 scope.go:122] "RemoveContainer" containerID="3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"
Nov 02 13:44:54.412430 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.412411    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"} err="failed to get container status \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": rpc error: code = NotFound desc = could not find container \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": container with ID starting with 3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d not found: ID does not exist"
Nov 02 13:44:54.412502 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.412431    6284 scope.go:122] "RemoveContainer" containerID="4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"
Nov 02 13:44:54.412646 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.412627    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"} err="failed to get container status \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": rpc error: code = NotFound desc = could not find container \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": container with ID starting with 4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b not found: ID does not exist"
Nov 02 13:44:54.412698 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.412647    6284 scope.go:122] "RemoveContainer" containerID="6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"
Nov 02 13:44:54.412830 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.412812    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"} err="failed to get container status \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": rpc error: code = NotFound desc = could not find container \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": container with ID starting with 6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748 not found: ID does not exist"
Nov 02 13:44:54.412880 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.412831    6284 scope.go:122] "RemoveContainer" containerID="39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"
Nov 02 13:44:54.413090 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.413073    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"} err="failed to get container status \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": rpc error: code = NotFound desc = could not find container \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": container with ID starting with 39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a not found: ID does not exist"
Nov 02 13:44:54.413131 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.413091    6284 scope.go:122] "RemoveContainer" containerID="e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"
Nov 02 13:44:54.413305 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.413289    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"} err="failed to get container status \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": rpc error: code = NotFound desc = could not find container \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": container with ID starting with e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504 not found: ID does not exist"
Nov 02 13:44:54.413344 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.413305    6284 scope.go:122] "RemoveContainer" containerID="273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"
Nov 02 13:44:54.413514 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.413499    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"} err="failed to get container status \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": rpc error: code = NotFound desc = could not find container \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": container with ID starting with 273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e not found: ID does not exist"
Nov 02 13:44:54.413565 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.413514    6284 scope.go:122] "RemoveContainer" containerID="9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"
Nov 02 13:44:54.413740 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.413723    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"} err="failed to get container status \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": rpc error: code = NotFound desc = could not find container \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": container with ID starting with 9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591 not found: ID does not exist"
Nov 02 13:44:54.413740 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.413738    6284 scope.go:122] "RemoveContainer" containerID="3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"
Nov 02 13:44:54.414012 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.413992    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"} err="failed to get container status \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": rpc error: code = NotFound desc = could not find container \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": container with ID starting with 3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d not found: ID does not exist"
Nov 02 13:44:54.414012 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.414012    6284 scope.go:122] "RemoveContainer" containerID="4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"
Nov 02 13:44:54.414248 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.414226    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"} err="failed to get container status \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": rpc error: code = NotFound desc = could not find container \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": container with ID starting with 4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b not found: ID does not exist"
Nov 02 13:44:54.414248 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.414248    6284 scope.go:122] "RemoveContainer" containerID="6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"
Nov 02 13:44:54.414450 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.414434    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"} err="failed to get container status \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": rpc error: code = NotFound desc = could not find container \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": container with ID starting with 6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748 not found: ID does not exist"
Nov 02 13:44:54.414450 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.414448    6284 scope.go:122] "RemoveContainer" containerID="39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"
Nov 02 13:44:54.414667 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.414650    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"} err="failed to get container status \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": rpc error: code = NotFound desc = could not find container \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": container with ID starting with 39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a not found: ID does not exist"
Nov 02 13:44:54.414711 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.414667    6284 scope.go:122] "RemoveContainer" containerID="e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"
Nov 02 13:44:54.414867 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.414851    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"} err="failed to get container status \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": rpc error: code = NotFound desc = could not find container \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": container with ID starting with e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504 not found: ID does not exist"
Nov 02 13:44:54.414910 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.414867    6284 scope.go:122] "RemoveContainer" containerID="273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"
Nov 02 13:44:54.415066 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.415051    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e"} err="failed to get container status \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": rpc error: code = NotFound desc = could not find container \"273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e\": container with ID starting with 273e87be91d1a69e49f27fbda4672533883639c834b2f37a84bc87ce63b0ae0e not found: ID does not exist"
Nov 02 13:44:54.415114 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.415066    6284 scope.go:122] "RemoveContainer" containerID="9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"
Nov 02 13:44:54.415263 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.415248    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591"} err="failed to get container status \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": rpc error: code = NotFound desc = could not find container \"9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591\": container with ID starting with 9ac529e4289e905dca9340273fce2bc071b49fadd8f1d4efbb9e053cdc322591 not found: ID does not exist"
Nov 02 13:44:54.415307 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.415263    6284 scope.go:122] "RemoveContainer" containerID="3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"
Nov 02 13:44:54.415456 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.415439    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d"} err="failed to get container status \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": rpc error: code = NotFound desc = could not find container \"3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d\": container with ID starting with 3bb1d0115e76f0b7938836b386b6069c47c4b6e7b7a45ab41aa244782e69c88d not found: ID does not exist"
Nov 02 13:44:54.415503 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.415456    6284 scope.go:122] "RemoveContainer" containerID="4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"
Nov 02 13:44:54.415669 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.415655    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b"} err="failed to get container status \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": rpc error: code = NotFound desc = could not find container \"4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b\": container with ID starting with 4e598ac35e7ff058d5b2a90fd78e37b74c40235b060e696cabba36f5298de45b not found: ID does not exist"
Nov 02 13:44:54.415714 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.415669    6284 scope.go:122] "RemoveContainer" containerID="6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"
Nov 02 13:44:54.415875 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.415852    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748"} err="failed to get container status \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": rpc error: code = NotFound desc = could not find container \"6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748\": container with ID starting with 6c5679054821f0c24ce37a2fafcc3be9b3256528f293bbf95ef91470488e8748 not found: ID does not exist"
Nov 02 13:44:54.415917 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.415875    6284 scope.go:122] "RemoveContainer" containerID="39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"
Nov 02 13:44:54.416115 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.416098    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a"} err="failed to get container status \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": rpc error: code = NotFound desc = could not find container \"39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a\": container with ID starting with 39cbc7ddc6bf6c8fe8a7b49c792be2a9422608808b3942bdaadf27d97dcee12a not found: ID does not exist"
Nov 02 13:44:54.416115 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.416114    6284 scope.go:122] "RemoveContainer" containerID="e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"
Nov 02 13:44:54.416335 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.416314    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504"} err="failed to get container status \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": rpc error: code = NotFound desc = could not find container \"e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504\": container with ID starting with e9ae3eaff550b4fa1d68551d00868f88c74543138c95de462ee5dcb691593504 not found: ID does not exist"
Nov 02 13:44:54.418592 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.418568    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 13:44:54.445423 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445339    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 13:44:54.445658 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445643    6284 cpu_manager.go:414] "containerMap: removing container" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy-web"
Nov 02 13:44:54.445695 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445660    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy-web"
Nov 02 13:44:54.445695 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445673    6284 cpu_manager.go:414] "containerMap: removing container" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="thanos-sidecar"
Nov 02 13:44:54.445695 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445678    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="thanos-sidecar"
Nov 02 13:44:54.445695 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445687    6284 cpu_manager.go:414] "containerMap: removing container" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="init-config-reloader"
Nov 02 13:44:54.445695 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445692    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="init-config-reloader"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445701    6284 cpu_manager.go:414] "containerMap: removing container" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445705    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445721    6284 cpu_manager.go:414] "containerMap: removing container" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="config-reloader"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445726    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="config-reloader"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445731    6284 cpu_manager.go:414] "containerMap: removing container" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="prometheus"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445736    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="prometheus"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445746    6284 cpu_manager.go:414] "containerMap: removing container" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy-thanos"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445750    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy-thanos"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445800    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy-web"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445810    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="thanos-sidecar"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445818    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="init-config-reloader"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445824    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445831    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="config-reloader"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445836    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="prometheus"
Nov 02 13:44:54.445844 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.445843    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" containerName="kube-rbac-proxy-thanos"
Nov 02 13:44:54.452226 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.452199    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.465918 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.465879    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 13:44:54.466138 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.466116    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 13:44:54.466191 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.466121    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 13:44:54.476724 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.476692    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 13:44:54.476961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.476746    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 13:44:54.476961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.476797    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 13:44:54.476961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.476836    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 13:44:54.476961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.476851    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 13:44:54.476961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.476858    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 13:44:54.476961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.476862    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 13:44:54.477263 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.477010    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7our0gqq5qckd\""
Nov 02 13:44:54.477263 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.477099    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 13:44:54.477263 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.477119    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 13:44:54.477263 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.477167    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xv47k\""
Nov 02 13:44:54.477441 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.477323    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 13:44:54.477540 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.477523    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 13:44:54.479639 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.479615    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 13:44:54.503098 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503033    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-web-config\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503113    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503142    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503164    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503230    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503255    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503288    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503387    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503427    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503449    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503468    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.503559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503489    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkp99\" (UniqueName: \"kubernetes.io/projected/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-kube-api-access-wkp99\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.504020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503607    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.504020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503644    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-config-out\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.504020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503673    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.504020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503712    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.504020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503742    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.504020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503773    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.504020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503794    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-config\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.504020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.503845    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604386 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604338    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604386 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604388    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604661 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604417    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604661 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604442    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604661 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604464    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604661 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604496    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604661 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604584    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604661 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604649    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604966 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604678    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604966 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604721    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604966 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604750    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkp99\" (UniqueName: \"kubernetes.io/projected/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-kube-api-access-wkp99\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604966 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604812    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604966 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604840    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-config-out\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604966 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604879    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.604966 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604927    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.605848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.604989    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.605848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.605014    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.605848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.605038    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-config\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.605848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.605064    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.605848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.605103    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-web-config\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.605848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.605324    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.605848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.605441    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.606218 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.605977    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.606606 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.606579    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.608706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.607303    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.608706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.607802    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-web-config\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.608706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.607897    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.608706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.608214    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.608706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.608540    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.608706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.608630    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.609115 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.608921    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-config\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.609317 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.609295    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.609701 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.609674    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.609847 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.609775    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.610000 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.609980    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.610202 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.610175    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-config-out\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.610430 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.610409    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.610752 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.610736    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.611549 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.611532    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.629478 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.629449    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkp99\" (UniqueName: \"kubernetes.io/projected/5cf8d1f5-65fc-4bdc-98d7-9b3732996772-kube-api-access-wkp99\") pod \"prometheus-k8s-0\" (UID: \"5cf8d1f5-65fc-4bdc-98d7-9b3732996772\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:54.780622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:54.780587    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:44:55.116660 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:55.116575    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 13:44:55.121161 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:44:55.121131    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf8d1f5_65fc_4bdc_98d7_9b3732996772.slice/crio-2fb0b17d543b12c23f1a9173ef905ad94aff95ac2018980b1a260b560194328e WatchSource:0}: Error finding container 2fb0b17d543b12c23f1a9173ef905ad94aff95ac2018980b1a260b560194328e: Status 404 returned error can't find the container with id 2fb0b17d543b12c23f1a9173ef905ad94aff95ac2018980b1a260b560194328e
Nov 02 13:44:55.355420 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:55.355372    6284 generic.go:356] "Generic (PLEG): container finished" podID="5cf8d1f5-65fc-4bdc-98d7-9b3732996772" containerID="d90266d27e17799116b4a51b9027114bb51b54edf092ec1c6cc9696ea5362ba0" exitCode=0
Nov 02 13:44:55.355598 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:55.355443    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cf8d1f5-65fc-4bdc-98d7-9b3732996772","Type":"ContainerDied","Data":"d90266d27e17799116b4a51b9027114bb51b54edf092ec1c6cc9696ea5362ba0"}
Nov 02 13:44:55.355598 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:55.355533    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cf8d1f5-65fc-4bdc-98d7-9b3732996772","Type":"ContainerStarted","Data":"2fb0b17d543b12c23f1a9173ef905ad94aff95ac2018980b1a260b560194328e"}
Nov 02 13:44:56.166908 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:56.166874    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da5bed06-f76d-474a-88ed-1fc6be553533" path="/var/lib/kubelet/pods/da5bed06-f76d-474a-88ed-1fc6be553533/volumes"
Nov 02 13:44:56.362469 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:56.362427    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cf8d1f5-65fc-4bdc-98d7-9b3732996772","Type":"ContainerStarted","Data":"3d75e35ba33758b0d0d1d7a201623d9a8f99bf9c01dd054c2382eac57fea07a8"}
Nov 02 13:44:56.362629 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:56.362493    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cf8d1f5-65fc-4bdc-98d7-9b3732996772","Type":"ContainerStarted","Data":"f748ac451a61784c4c82baccf98c9bc732c8335f1df15d20b0dca4e051cf0123"}
Nov 02 13:44:56.362629 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:56.362520    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cf8d1f5-65fc-4bdc-98d7-9b3732996772","Type":"ContainerStarted","Data":"250b3d24e3e3da06041ff1a474099823d5d20c78c5cb62ca375b649694be7a85"}
Nov 02 13:44:56.362629 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:56.362548    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cf8d1f5-65fc-4bdc-98d7-9b3732996772","Type":"ContainerStarted","Data":"c3cffa94a4c5c6c0451b819f40784ce7b47459979e93045e74d2187a1d3f8772"}
Nov 02 13:44:56.362629 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:56.362576    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cf8d1f5-65fc-4bdc-98d7-9b3732996772","Type":"ContainerStarted","Data":"cb5bc95d454ab186f359268293b2eda9313d3fd37a08bfd863b0100b5511e35b"}
Nov 02 13:44:56.362629 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:56.362600    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5cf8d1f5-65fc-4bdc-98d7-9b3732996772","Type":"ContainerStarted","Data":"e5a7129dfc462ed8027980be515e5fc4fee9df7c4b857dcd7ef8b24ae98a360f"}
Nov 02 13:44:56.424712 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:56.424611    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.40799308 podStartE2EDuration="2.424597743s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.604663ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:56.423597946 +0000 UTC m=+28.382973553" watchObservedRunningTime="2026-11-02 13:44:56.424597743 +0000 UTC m=+28.383973350"
Nov 02 13:44:59.781309 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:44:59.781264    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:45:00.241456 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:45:00.241430    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-57d72"
Nov 02 13:45:13.893772 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:45:13.893731    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:45:39.419161 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:45:39.419106    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:45:39.434990 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:45:39.434963    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:45:39.533174 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:45:39.533146    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 13:45:43.893416 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:45:43.893370    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:46:13.892696 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:46:13.892661    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:46:43.892189 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:46:43.892141    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:47:13.891681 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:47:13.891642    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:47:43.891808 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:47:43.891730    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:47:54.870362 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:54.870330    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"]
Nov 02 13:47:54.873722 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:54.873701    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:54.900478 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:54.900457    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-sbgqp\""
Nov 02 13:47:54.900645 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:54.900627    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 13:47:54.900767 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:54.900703    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 13:47:54.901418 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:54.901400    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"]
Nov 02 13:47:54.956794 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:54.956773    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:54.956921 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:54.956827    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsmjw\" (UniqueName: \"kubernetes.io/projected/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-kube-api-access-zsmjw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:54.957009 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:54.956988    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:55.058100 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:55.058066    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsmjw\" (UniqueName: \"kubernetes.io/projected/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-kube-api-access-zsmjw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:55.058231 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:55.058127    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:55.058231 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:55.058192    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:55.058516 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:55.058496    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:55.058589 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:55.058519    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:55.081301 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:55.081280    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsmjw\" (UniqueName: \"kubernetes.io/projected/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-kube-api-access-zsmjw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:55.188773 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:55.188724    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:47:55.481691 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:55.481590    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"]
Nov 02 13:47:55.485973 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:47:55.485883    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8bd1e4_e1d6_4ad4_9170_8ce900fbc4fc.slice/crio-28b62215fbef6af1c37f6b32f13cab20eaccf3dd8b90c3debca8abda58c8fffe WatchSource:0}: Error finding container 28b62215fbef6af1c37f6b32f13cab20eaccf3dd8b90c3debca8abda58c8fffe: Status 404 returned error can't find the container with id 28b62215fbef6af1c37f6b32f13cab20eaccf3dd8b90c3debca8abda58c8fffe
Nov 02 13:47:55.488048 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:55.488029    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:47:55.977323 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:55.977293    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl" event={"ID":"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc","Type":"ContainerStarted","Data":"28b62215fbef6af1c37f6b32f13cab20eaccf3dd8b90c3debca8abda58c8fffe"}
Nov 02 13:47:56.610435 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.610407    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"]
Nov 02 13:47:56.614083 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.614054    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.636920 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.636884    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-g4x97\""
Nov 02 13:47:56.637870 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.637838    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"]
Nov 02 13:47:56.773574 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.773539    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.773574 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.773585    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.773853 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.773755    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szn76\" (UniqueName: \"kubernetes.io/projected/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-kube-api-access-szn76\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.818277 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.818253    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"]
Nov 02 13:47:56.822040 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.822020    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:56.834445 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.834422    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-mktzc\""
Nov 02 13:47:56.843778 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.843760    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"]
Nov 02 13:47:56.875230 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.875168    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szn76\" (UniqueName: \"kubernetes.io/projected/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-kube-api-access-szn76\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.875230 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.875213    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.875424 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.875233    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.875643 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.875621    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.875643 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.875635    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.896441 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.896412    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szn76\" (UniqueName: \"kubernetes.io/projected/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-kube-api-access-szn76\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.922475 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.922448    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"]
Nov 02 13:47:56.926056 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.926039    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:56.931856 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.931831    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:47:56.938179 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.938159    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-l4vxc\""
Nov 02 13:47:56.951688 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.951664    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"]
Nov 02 13:47:56.976188 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.976163    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq97b\" (UniqueName: \"kubernetes.io/projected/ecd5fcd3-26b4-432b-b8d5-72155727b96c-kube-api-access-fq97b\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:56.976325 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.976208    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:56.976325 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:56.976270    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:57.016190 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.016165    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"]
Nov 02 13:47:57.020427 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.020355    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.032762 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.032741    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-bjkn8\""
Nov 02 13:47:57.038517 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.038491    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"]
Nov 02 13:47:57.076753 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.076716    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq97b\" (UniqueName: \"kubernetes.io/projected/ecd5fcd3-26b4-432b-b8d5-72155727b96c-kube-api-access-fq97b\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:57.076930 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.076778    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:57.076930 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.076809    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:57.076930 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.076833    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:57.076930 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.076880    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:57.076930 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.076910    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q64xq\" (UniqueName: \"kubernetes.io/projected/1605600e-1f07-47d6-b378-6de78a500bc0-kube-api-access-q64xq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:57.077658 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.077632    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:57.077773 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.077697    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:57.098290 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.098263    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq97b\" (UniqueName: \"kubernetes.io/projected/ecd5fcd3-26b4-432b-b8d5-72155727b96c-kube-api-access-fq97b\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:57.118611 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.118584    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"]
Nov 02 13:47:57.122814 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.122791    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.135452 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.135425    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-nbl9q\""
Nov 02 13:47:57.138194 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.138173    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:47:57.140742 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.140724    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"]
Nov 02 13:47:57.177671 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.177636    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.177825 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.177700    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jktvf\" (UniqueName: \"kubernetes.io/projected/70acb887-3146-4f03-97cc-19904033a3dc-kube-api-access-jktvf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.177904 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.177819    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:57.177904 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.177869    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:57.178063 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.177965    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q64xq\" (UniqueName: \"kubernetes.io/projected/1605600e-1f07-47d6-b378-6de78a500bc0-kube-api-access-q64xq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:57.178063 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.178005    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.178367 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.178344    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:57.178436 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.178365    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:57.202550 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.202533    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q64xq\" (UniqueName: \"kubernetes.io/projected/1605600e-1f07-47d6-b378-6de78a500bc0-kube-api-access-q64xq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:57.230015 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.229981    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"]
Nov 02 13:47:57.236269 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:47:57.236227    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5143ef_5f0f_48e9_8e1b_6aa388d7101a.slice/crio-1790942b19e3a72afc48856b1c18c795a78aa9094dbe663acc6b5963c85ae358 WatchSource:0}: Error finding container 1790942b19e3a72afc48856b1c18c795a78aa9094dbe663acc6b5963c85ae358: Status 404 returned error can't find the container with id 1790942b19e3a72afc48856b1c18c795a78aa9094dbe663acc6b5963c85ae358
Nov 02 13:47:57.242060 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.242041    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:47:57.279198 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.279168    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2q9\" (UniqueName: \"kubernetes.io/projected/8161d4b7-b679-43c2-bed4-d5db6256b9c9-kube-api-access-fm2q9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.279331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.279235    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.279331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.279262    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.279331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.279280    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.279331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.279307    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.279331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.279325    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jktvf\" (UniqueName: \"kubernetes.io/projected/70acb887-3146-4f03-97cc-19904033a3dc-kube-api-access-jktvf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.280297 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.280274    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.280450 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.280350    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.306452 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.306409    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jktvf\" (UniqueName: \"kubernetes.io/projected/70acb887-3146-4f03-97cc-19904033a3dc-kube-api-access-jktvf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.341834 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.341803    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:47:57.380298 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.379271    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"]
Nov 02 13:47:57.381338 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.380782    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2q9\" (UniqueName: \"kubernetes.io/projected/8161d4b7-b679-43c2-bed4-d5db6256b9c9-kube-api-access-fm2q9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.381338 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.380843    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.381338 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.380873    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.381338 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.381217    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.381338 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.381289    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.386083 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.386032    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.410378 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.407244    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-ljz6f\""
Nov 02 13:47:57.410378 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.408250    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"]
Nov 02 13:47:57.410378 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.409152    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2q9\" (UniqueName: \"kubernetes.io/projected/8161d4b7-b679-43c2-bed4-d5db6256b9c9-kube-api-access-fm2q9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.443821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.443787    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:47:57.448146 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.448076    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"]
Nov 02 13:47:57.453410 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:47:57.453379    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd5fcd3_26b4_432b_b8d5_72155727b96c.slice/crio-35ae17284a5b24a34312d773016b48eb22ba83205558a43394e7303c7c2144d9 WatchSource:0}: Error finding container 35ae17284a5b24a34312d773016b48eb22ba83205558a43394e7303c7c2144d9: Status 404 returned error can't find the container with id 35ae17284a5b24a34312d773016b48eb22ba83205558a43394e7303c7c2144d9
Nov 02 13:47:57.481396 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.481371    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.481555 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.481536    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.481627 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.481564    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4kz\" (UniqueName: \"kubernetes.io/projected/ac90c685-6134-4ccc-83c4-01f57629a4f2-kube-api-access-8z4kz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.543614 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.543588    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"]
Nov 02 13:47:57.547836 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:47:57.547810    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1605600e_1f07_47d6_b378_6de78a500bc0.slice/crio-47e426bfd36e805421cee0c5e69eaf666a0cf337d246adeddaf837d6ee61517d WatchSource:0}: Error finding container 47e426bfd36e805421cee0c5e69eaf666a0cf337d246adeddaf837d6ee61517d: Status 404 returned error can't find the container with id 47e426bfd36e805421cee0c5e69eaf666a0cf337d246adeddaf837d6ee61517d
Nov 02 13:47:57.582547 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.582516    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.582676 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.582562    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4kz\" (UniqueName: \"kubernetes.io/projected/ac90c685-6134-4ccc-83c4-01f57629a4f2-kube-api-access-8z4kz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.582676 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.582596    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.583026 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.583002    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.583138 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.583067    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.604481 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.604443    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4kz\" (UniqueName: \"kubernetes.io/projected/ac90c685-6134-4ccc-83c4-01f57629a4f2-kube-api-access-8z4kz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.642238 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.642162    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"]
Nov 02 13:47:57.648191 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:47:57.648160    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70acb887_3146_4f03_97cc_19904033a3dc.slice/crio-4c96e7a7520a5f0a7df3696362043dda82f1e3f97ddb63af95abe12b9ecd585c WatchSource:0}: Error finding container 4c96e7a7520a5f0a7df3696362043dda82f1e3f97ddb63af95abe12b9ecd585c: Status 404 returned error can't find the container with id 4c96e7a7520a5f0a7df3696362043dda82f1e3f97ddb63af95abe12b9ecd585c
Nov 02 13:47:57.721574 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.721537    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:47:57.746688 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.746660    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"]
Nov 02 13:47:57.756069 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:47:57.755997    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8161d4b7_b679_43c2_bed4_d5db6256b9c9.slice/crio-b901e01e9ddcaf308cad24779ad97e4de18857f4ddf3e11a0dedf5e8abe79353 WatchSource:0}: Error finding container b901e01e9ddcaf308cad24779ad97e4de18857f4ddf3e11a0dedf5e8abe79353: Status 404 returned error can't find the container with id b901e01e9ddcaf308cad24779ad97e4de18857f4ddf3e11a0dedf5e8abe79353
Nov 02 13:47:57.992712 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.992631    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk" event={"ID":"8161d4b7-b679-43c2-bed4-d5db6256b9c9","Type":"ContainerStarted","Data":"b901e01e9ddcaf308cad24779ad97e4de18857f4ddf3e11a0dedf5e8abe79353"}
Nov 02 13:47:57.999744 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:57.999714    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c" event={"ID":"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a","Type":"ContainerStarted","Data":"1790942b19e3a72afc48856b1c18c795a78aa9094dbe663acc6b5963c85ae358"}
Nov 02 13:47:58.001467 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:58.001440    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm" event={"ID":"ecd5fcd3-26b4-432b-b8d5-72155727b96c","Type":"ContainerStarted","Data":"35ae17284a5b24a34312d773016b48eb22ba83205558a43394e7303c7c2144d9"}
Nov 02 13:47:58.002964 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:58.002907    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk" event={"ID":"70acb887-3146-4f03-97cc-19904033a3dc","Type":"ContainerStarted","Data":"4c96e7a7520a5f0a7df3696362043dda82f1e3f97ddb63af95abe12b9ecd585c"}
Nov 02 13:47:58.004449 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:58.004427    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k" event={"ID":"1605600e-1f07-47d6-b378-6de78a500bc0","Type":"ContainerStarted","Data":"47e426bfd36e805421cee0c5e69eaf666a0cf337d246adeddaf837d6ee61517d"}
Nov 02 13:47:58.030171 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:58.030145    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"]
Nov 02 13:47:58.039611 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:47:58.039576    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac90c685_6134_4ccc_83c4_01f57629a4f2.slice/crio-00cdaa7c7e434f065429e870f2674e14affb5963a2f46f5e013d6c0362033b4e WatchSource:0}: Error finding container 00cdaa7c7e434f065429e870f2674e14affb5963a2f46f5e013d6c0362033b4e: Status 404 returned error can't find the container with id 00cdaa7c7e434f065429e870f2674e14affb5963a2f46f5e013d6c0362033b4e
Nov 02 13:47:59.022349 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:47:59.022275    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x" event={"ID":"ac90c685-6134-4ccc-83c4-01f57629a4f2","Type":"ContainerStarted","Data":"00cdaa7c7e434f065429e870f2674e14affb5963a2f46f5e013d6c0362033b4e"}
Nov 02 13:48:08.095264 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:08.095231    6284 generic.go:356] "Generic (PLEG): container finished" podID="5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" containerID="4a03a88d868ad83ec950d1c801d82e1ad6cf781054d70888b5d190a82db463aa" exitCode=0
Nov 02 13:48:08.095682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:08.095313    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl" event={"ID":"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc","Type":"ContainerDied","Data":"4a03a88d868ad83ec950d1c801d82e1ad6cf781054d70888b5d190a82db463aa"}
Nov 02 13:48:11.120614 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:11.119490    6284 generic.go:356] "Generic (PLEG): container finished" podID="9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" containerID="8e3f7ff988667fbb4b9ebac5431fcc7304a88dbe448f6a52d3adf6b72d6bb8ce" exitCode=0
Nov 02 13:48:11.120614 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:11.119552    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c" event={"ID":"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a","Type":"ContainerDied","Data":"8e3f7ff988667fbb4b9ebac5431fcc7304a88dbe448f6a52d3adf6b72d6bb8ce"}
Nov 02 13:48:12.125019 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:12.124980    6284 generic.go:356] "Generic (PLEG): container finished" podID="ecd5fcd3-26b4-432b-b8d5-72155727b96c" containerID="6af23c87c60d09aa54b4b1074a9e0affc4f28231c8bc60ffddc640dfb203efa4" exitCode=0
Nov 02 13:48:12.125422 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:12.125065    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm" event={"ID":"ecd5fcd3-26b4-432b-b8d5-72155727b96c","Type":"ContainerDied","Data":"6af23c87c60d09aa54b4b1074a9e0affc4f28231c8bc60ffddc640dfb203efa4"}
Nov 02 13:48:12.126594 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:12.126574    6284 generic.go:356] "Generic (PLEG): container finished" podID="1605600e-1f07-47d6-b378-6de78a500bc0" containerID="61beaa014cfdef9cf9c90ae046f0e9cd5f7617774ce35d1b816312147dee64f0" exitCode=0
Nov 02 13:48:12.126682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:12.126616    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k" event={"ID":"1605600e-1f07-47d6-b378-6de78a500bc0","Type":"ContainerDied","Data":"61beaa014cfdef9cf9c90ae046f0e9cd5f7617774ce35d1b816312147dee64f0"}
Nov 02 13:48:13.132169 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.132144    6284 generic.go:356] "Generic (PLEG): container finished" podID="ecd5fcd3-26b4-432b-b8d5-72155727b96c" containerID="15728ef20cfcb096749eb38a9c704666e5de4963245fc8cca9f672f6c9b0f68f" exitCode=0
Nov 02 13:48:13.132539 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.132227    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm" event={"ID":"ecd5fcd3-26b4-432b-b8d5-72155727b96c","Type":"ContainerDied","Data":"15728ef20cfcb096749eb38a9c704666e5de4963245fc8cca9f672f6c9b0f68f"}
Nov 02 13:48:13.133850 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.133820    6284 generic.go:356] "Generic (PLEG): container finished" podID="70acb887-3146-4f03-97cc-19904033a3dc" containerID="40bca909975846fbe145464894ef356e4bd8fb04b8f4d8bd9be7c82c1eda41e2" exitCode=0
Nov 02 13:48:13.133970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.133861    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk" event={"ID":"70acb887-3146-4f03-97cc-19904033a3dc","Type":"ContainerDied","Data":"40bca909975846fbe145464894ef356e4bd8fb04b8f4d8bd9be7c82c1eda41e2"}
Nov 02 13:48:13.135852 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.135829    6284 generic.go:356] "Generic (PLEG): container finished" podID="5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" containerID="962ce107c9235dbff24f7a3b9ba6c64319b53f4f4713330a7f899f074d40c1a9" exitCode=0
Nov 02 13:48:13.135955 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.135855    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl" event={"ID":"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc","Type":"ContainerDied","Data":"962ce107c9235dbff24f7a3b9ba6c64319b53f4f4713330a7f899f074d40c1a9"}
Nov 02 13:48:13.138462 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.138439    6284 generic.go:356] "Generic (PLEG): container finished" podID="1605600e-1f07-47d6-b378-6de78a500bc0" containerID="211b079aa6d50ac8d55030dc26a2518b96dd442de58ecc78885b5399c71dc589" exitCode=0
Nov 02 13:48:13.138563 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.138522    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k" event={"ID":"1605600e-1f07-47d6-b378-6de78a500bc0","Type":"ContainerDied","Data":"211b079aa6d50ac8d55030dc26a2518b96dd442de58ecc78885b5399c71dc589"}
Nov 02 13:48:13.140238 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.140218    6284 generic.go:356] "Generic (PLEG): container finished" podID="8161d4b7-b679-43c2-bed4-d5db6256b9c9" containerID="84efef39898dc3d60e8c2daf9acb21ed3441a5faf834b2189faa511585046b00" exitCode=0
Nov 02 13:48:13.140340 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.140278    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk" event={"ID":"8161d4b7-b679-43c2-bed4-d5db6256b9c9","Type":"ContainerDied","Data":"84efef39898dc3d60e8c2daf9acb21ed3441a5faf834b2189faa511585046b00"}
Nov 02 13:48:13.142220 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.142202    6284 generic.go:356] "Generic (PLEG): container finished" podID="9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" containerID="e77791571589a8ef5ddfbd747b0b41f827c927c132b56ec9d81e1af87ceaf7db" exitCode=0
Nov 02 13:48:13.142305 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.142288    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c" event={"ID":"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a","Type":"ContainerDied","Data":"e77791571589a8ef5ddfbd747b0b41f827c927c132b56ec9d81e1af87ceaf7db"}
Nov 02 13:48:13.144054 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.144028    6284 generic.go:356] "Generic (PLEG): container finished" podID="ac90c685-6134-4ccc-83c4-01f57629a4f2" containerID="27cdd1fae12d21eb86dc02b4e8901aab15d6ce509ef6f63114ae7644e1f4062e" exitCode=0
Nov 02 13:48:13.144162 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:13.144055    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x" event={"ID":"ac90c685-6134-4ccc-83c4-01f57629a4f2","Type":"ContainerDied","Data":"27cdd1fae12d21eb86dc02b4e8901aab15d6ce509ef6f63114ae7644e1f4062e"}
Nov 02 13:48:13.903356 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:48:13.903271    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:48:14.153517 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:14.153319    6284 generic.go:356] "Generic (PLEG): container finished" podID="70acb887-3146-4f03-97cc-19904033a3dc" containerID="caf1960419380d69fe0edea48b7943592daaf16d58c844206b9a6e87ff792f25" exitCode=0
Nov 02 13:48:14.153517 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:14.153445    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk" event={"ID":"70acb887-3146-4f03-97cc-19904033a3dc","Type":"ContainerDied","Data":"caf1960419380d69fe0edea48b7943592daaf16d58c844206b9a6e87ff792f25"}
Nov 02 13:48:14.158526 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:14.157855    6284 generic.go:356] "Generic (PLEG): container finished" podID="8161d4b7-b679-43c2-bed4-d5db6256b9c9" containerID="7dbef768b93905a81af4530fde2ff20ed9f773df119dc175e78d957db940ced8" exitCode=0
Nov 02 13:48:14.158871 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:14.158724    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk" event={"ID":"8161d4b7-b679-43c2-bed4-d5db6256b9c9","Type":"ContainerDied","Data":"7dbef768b93905a81af4530fde2ff20ed9f773df119dc175e78d957db940ced8"}
Nov 02 13:48:18.209705 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:18.209345    6284 generic.go:356] "Generic (PLEG): container finished" podID="ac90c685-6134-4ccc-83c4-01f57629a4f2" containerID="b9361af1685c526321b0b356276c9ddf760ecbd494d26f7510e5054754382fc2" exitCode=0
Nov 02 13:48:18.209705 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:18.209409    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x" event={"ID":"ac90c685-6134-4ccc-83c4-01f57629a4f2","Type":"ContainerDied","Data":"b9361af1685c526321b0b356276c9ddf760ecbd494d26f7510e5054754382fc2"}
Nov 02 13:48:28.296971 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:28.296919    6284 generic.go:356] "Generic (PLEG): container finished" podID="9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" containerID="bcaf4eed58e930f9b0a94253418a17d831d5ad418ffba8f5b1d47d4e935a49ff" exitCode=0
Nov 02 13:48:28.297526 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:28.296976    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c" event={"ID":"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a","Type":"ContainerDied","Data":"bcaf4eed58e930f9b0a94253418a17d831d5ad418ffba8f5b1d47d4e935a49ff"}
Nov 02 13:48:28.299682 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:28.299655    6284 generic.go:356] "Generic (PLEG): container finished" podID="ac90c685-6134-4ccc-83c4-01f57629a4f2" containerID="01de427ed18a77f37dc0487a481e98e0e897cca45220e0b3b2bf14105bc242b8" exitCode=0
Nov 02 13:48:28.299803 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:28.299747    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x" event={"ID":"ac90c685-6134-4ccc-83c4-01f57629a4f2","Type":"ContainerDied","Data":"01de427ed18a77f37dc0487a481e98e0e897cca45220e0b3b2bf14105bc242b8"}
Nov 02 13:48:28.302158 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:28.302136    6284 generic.go:356] "Generic (PLEG): container finished" podID="ecd5fcd3-26b4-432b-b8d5-72155727b96c" containerID="76976b1269a323c58defe17db74ddd897bda7a3f085157d52ddacd2d3ae47e99" exitCode=0
Nov 02 13:48:28.302264 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:28.302219    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm" event={"ID":"ecd5fcd3-26b4-432b-b8d5-72155727b96c","Type":"ContainerDied","Data":"76976b1269a323c58defe17db74ddd897bda7a3f085157d52ddacd2d3ae47e99"}
Nov 02 13:48:28.304618 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:28.304598    6284 generic.go:356] "Generic (PLEG): container finished" podID="5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" containerID="ad4d88a25553f7a299abf4aba57ff6a24e50e4639fe71925717d51a7b13434d0" exitCode=0
Nov 02 13:48:28.304688 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:28.304669    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl" event={"ID":"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc","Type":"ContainerDied","Data":"ad4d88a25553f7a299abf4aba57ff6a24e50e4639fe71925717d51a7b13434d0"}
Nov 02 13:48:28.307166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:28.307146    6284 generic.go:356] "Generic (PLEG): container finished" podID="1605600e-1f07-47d6-b378-6de78a500bc0" containerID="e8e0c409fcf580c5fe3d03f73bcf781c510853ab10b1d458e96f548c3c1e388a" exitCode=0
Nov 02 13:48:28.307252 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:28.307174    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k" event={"ID":"1605600e-1f07-47d6-b378-6de78a500bc0","Type":"ContainerDied","Data":"e8e0c409fcf580c5fe3d03f73bcf781c510853ab10b1d458e96f548c3c1e388a"}
Nov 02 13:48:29.313542 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.313499    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk" event={"ID":"70acb887-3146-4f03-97cc-19904033a3dc","Type":"ContainerStarted","Data":"b399d37814314458f5c247d60a41aa14e0623c1789980ed4fb22008c971826c8"}
Nov 02 13:48:29.315735 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.315707    6284 generic.go:356] "Generic (PLEG): container finished" podID="8161d4b7-b679-43c2-bed4-d5db6256b9c9" containerID="9636e8022d9097b7aa88e160267a3d93f3ee3a9fbcd4003b29825712d532fe2a" exitCode=0
Nov 02 13:48:29.315879 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.315789    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk" event={"ID":"8161d4b7-b679-43c2-bed4-d5db6256b9c9","Type":"ContainerDied","Data":"9636e8022d9097b7aa88e160267a3d93f3ee3a9fbcd4003b29825712d532fe2a"}
Nov 02 13:48:29.356928 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.355451    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk" podStartSLOduration=3.343870827 podStartE2EDuration="33.355430245s" totalImagesPullingTime="30.011559418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:47:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:29.353497101 +0000 UTC m=+241.312872733" watchObservedRunningTime="2026-11-02 13:48:29.355430245 +0000 UTC m=+241.314805853"
Nov 02 13:48:29.459080 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.459054    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:48:29.492889 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.492858    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:48:29.538281 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.537999    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:48:29.542531 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.542512    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:48:29.546557 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.546533    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:48:29.611688 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.611626    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-kube-api-access-szn76\" (UniqueName: \"kubernetes.io/projected/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-kube-api-access-szn76\") pod \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") "
Nov 02 13:48:29.611688 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.611658    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac90c685-6134-4ccc-83c4-01f57629a4f2-kube-api-access-8z4kz\" (UniqueName: \"kubernetes.io/projected/ac90c685-6134-4ccc-83c4-01f57629a4f2-kube-api-access-8z4kz\") pod \"ac90c685-6134-4ccc-83c4-01f57629a4f2\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") "
Nov 02 13:48:29.611688 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.611686    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd5fcd3-26b4-432b-b8d5-72155727b96c-kube-api-access-fq97b\" (UniqueName: \"kubernetes.io/projected/ecd5fcd3-26b4-432b-b8d5-72155727b96c-kube-api-access-fq97b\") pod \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") "
Nov 02 13:48:29.611948 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.611731    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-util\" (UniqueName: \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-util\") pod \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") "
Nov 02 13:48:29.611948 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.611778    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-util\" (UniqueName: \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-util\") pod \"ac90c685-6134-4ccc-83c4-01f57629a4f2\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") "
Nov 02 13:48:29.611948 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.611811    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-bundle\") pod \"ac90c685-6134-4ccc-83c4-01f57629a4f2\" (UID: \"ac90c685-6134-4ccc-83c4-01f57629a4f2\") "
Nov 02 13:48:29.611948 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.611850    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-kube-api-access-zsmjw\" (UniqueName: \"kubernetes.io/projected/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-kube-api-access-zsmjw\") pod \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") "
Nov 02 13:48:29.611948 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.611899    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1605600e-1f07-47d6-b378-6de78a500bc0-kube-api-access-q64xq\" (UniqueName: \"kubernetes.io/projected/1605600e-1f07-47d6-b378-6de78a500bc0-kube-api-access-q64xq\") pod \"1605600e-1f07-47d6-b378-6de78a500bc0\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") "
Nov 02 13:48:29.612224 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.611969    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-bundle\") pod \"1605600e-1f07-47d6-b378-6de78a500bc0\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") "
Nov 02 13:48:29.612224 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.612002    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-bundle\") pod \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\" (UID: \"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc\") "
Nov 02 13:48:29.612224 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.612035    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-bundle\") pod \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") "
Nov 02 13:48:29.612224 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.612081    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-util\" (UniqueName: \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-util\") pod \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\" (UID: \"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a\") "
Nov 02 13:48:29.612224 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.612114    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-bundle\") pod \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") "
Nov 02 13:48:29.612224 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.612138    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-util\" (UniqueName: \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-util\") pod \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\" (UID: \"ecd5fcd3-26b4-432b-b8d5-72155727b96c\") "
Nov 02 13:48:29.612224 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.612173    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-util\" (UniqueName: \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-util\") pod \"1605600e-1f07-47d6-b378-6de78a500bc0\" (UID: \"1605600e-1f07-47d6-b378-6de78a500bc0\") "
Nov 02 13:48:29.616320 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.612546    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-bundle" pod "ac90c685-6134-4ccc-83c4-01f57629a4f2" (UID: "ac90c685-6134-4ccc-83c4-01f57629a4f2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:29.616320 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.612845    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-bundle" pod "1605600e-1f07-47d6-b378-6de78a500bc0" (UID: "1605600e-1f07-47d6-b378-6de78a500bc0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:29.616320 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.613498    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-bundle" pod "5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" (UID: "5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:29.616320 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.614368    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd5fcd3-26b4-432b-b8d5-72155727b96c-kube-api-access-fq97b" pod "ecd5fcd3-26b4-432b-b8d5-72155727b96c" (UID: "ecd5fcd3-26b4-432b-b8d5-72155727b96c"). InnerVolumeSpecName "kube-api-access-fq97b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:29.616320 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.614491    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-bundle" pod "9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" (UID: "9a5143ef-5f0f-48e9-8e1b-6aa388d7101a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:29.616320 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.614687    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-bundle" pod "ecd5fcd3-26b4-432b-b8d5-72155727b96c" (UID: "ecd5fcd3-26b4-432b-b8d5-72155727b96c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:29.616320 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.615915    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-kube-api-access-szn76" pod "9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" (UID: "9a5143ef-5f0f-48e9-8e1b-6aa388d7101a"). InnerVolumeSpecName "kube-api-access-szn76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:29.621210 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.618730    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1605600e-1f07-47d6-b378-6de78a500bc0-kube-api-access-q64xq" pod "1605600e-1f07-47d6-b378-6de78a500bc0" (UID: "1605600e-1f07-47d6-b378-6de78a500bc0"). InnerVolumeSpecName "kube-api-access-q64xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:29.621210 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.618925    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-kube-api-access-zsmjw" pod "5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" (UID: "5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc"). InnerVolumeSpecName "kube-api-access-zsmjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:29.621210 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.620844    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-util" pod "ac90c685-6134-4ccc-83c4-01f57629a4f2" (UID: "ac90c685-6134-4ccc-83c4-01f57629a4f2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:29.622314 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.622284    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-util" pod "1605600e-1f07-47d6-b378-6de78a500bc0" (UID: "1605600e-1f07-47d6-b378-6de78a500bc0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:29.622745 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.622721    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-util" pod "5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" (UID: "5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:29.623075 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.623047    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac90c685-6134-4ccc-83c4-01f57629a4f2-kube-api-access-8z4kz" pod "ac90c685-6134-4ccc-83c4-01f57629a4f2" (UID: "ac90c685-6134-4ccc-83c4-01f57629a4f2"). InnerVolumeSpecName "kube-api-access-8z4kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:29.623779 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.623756    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-util" pod "ecd5fcd3-26b4-432b-b8d5-72155727b96c" (UID: "ecd5fcd3-26b4-432b-b8d5-72155727b96c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:29.624381 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.624363    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-util" pod "9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" (UID: "9a5143ef-5f0f-48e9-8e1b-6aa388d7101a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:29.713533 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713504    6284 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-util\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713533 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713528    6284 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713533 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713537    6284 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ecd5fcd3-26b4-432b-b8d5-72155727b96c-util\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713544    6284 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-util\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713552    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szn76\" (UniqueName: \"kubernetes.io/projected/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-kube-api-access-szn76\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713561    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z4kz\" (UniqueName: \"kubernetes.io/projected/ac90c685-6134-4ccc-83c4-01f57629a4f2-kube-api-access-8z4kz\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713569    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq97b\" (UniqueName: \"kubernetes.io/projected/ecd5fcd3-26b4-432b-b8d5-72155727b96c-kube-api-access-fq97b\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713578    6284 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-util\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713587    6284 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-util\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713594    6284 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac90c685-6134-4ccc-83c4-01f57629a4f2-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713602    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsmjw\" (UniqueName: \"kubernetes.io/projected/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-kube-api-access-zsmjw\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713610    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q64xq\" (UniqueName: \"kubernetes.io/projected/1605600e-1f07-47d6-b378-6de78a500bc0-kube-api-access-q64xq\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713619    6284 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1605600e-1f07-47d6-b378-6de78a500bc0-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713627    6284 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:29.713741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:29.713634    6284 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5143ef-5f0f-48e9-8e1b-6aa388d7101a-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:30.321310 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.321278    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm" event={"ID":"ecd5fcd3-26b4-432b-b8d5-72155727b96c","Type":"ContainerDied","Data":"35ae17284a5b24a34312d773016b48eb22ba83205558a43394e7303c7c2144d9"}
Nov 02 13:48:30.321310 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.321293    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a49xhm"
Nov 02 13:48:30.321784 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.321325    6284 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ae17284a5b24a34312d773016b48eb22ba83205558a43394e7303c7c2144d9"
Nov 02 13:48:30.323596 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.323556    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl"
Nov 02 13:48:30.323596 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.323562    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c4flbl" event={"ID":"5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc","Type":"ContainerDied","Data":"28b62215fbef6af1c37f6b32f13cab20eaccf3dd8b90c3debca8abda58c8fffe"}
Nov 02 13:48:30.323757 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.323599    6284 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28b62215fbef6af1c37f6b32f13cab20eaccf3dd8b90c3debca8abda58c8fffe"
Nov 02 13:48:30.326084 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.326064    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k"
Nov 02 13:48:30.326179 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.326083    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jfb9k" event={"ID":"1605600e-1f07-47d6-b378-6de78a500bc0","Type":"ContainerDied","Data":"47e426bfd36e805421cee0c5e69eaf666a0cf337d246adeddaf837d6ee61517d"}
Nov 02 13:48:30.326179 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.326115    6284 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47e426bfd36e805421cee0c5e69eaf666a0cf337d246adeddaf837d6ee61517d"
Nov 02 13:48:30.328116 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.328098    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c"
Nov 02 13:48:30.328210 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.328098    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vf56c" event={"ID":"9a5143ef-5f0f-48e9-8e1b-6aa388d7101a","Type":"ContainerDied","Data":"1790942b19e3a72afc48856b1c18c795a78aa9094dbe663acc6b5963c85ae358"}
Nov 02 13:48:30.328264 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.328211    6284 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1790942b19e3a72afc48856b1c18c795a78aa9094dbe663acc6b5963c85ae358"
Nov 02 13:48:30.330258 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.330240    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x"
Nov 02 13:48:30.330345 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.330259    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442dw5x" event={"ID":"ac90c685-6134-4ccc-83c4-01f57629a4f2","Type":"ContainerDied","Data":"00cdaa7c7e434f065429e870f2674e14affb5963a2f46f5e013d6c0362033b4e"}
Nov 02 13:48:30.330345 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.330290    6284 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00cdaa7c7e434f065429e870f2674e14affb5963a2f46f5e013d6c0362033b4e"
Nov 02 13:48:30.446524 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.446503    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:48:30.521407 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.521376    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8161d4b7-b679-43c2-bed4-d5db6256b9c9-kube-api-access-fm2q9\" (UniqueName: \"kubernetes.io/projected/8161d4b7-b679-43c2-bed4-d5db6256b9c9-kube-api-access-fm2q9\") pod \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") "
Nov 02 13:48:30.521550 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.521439    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-bundle\") pod \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") "
Nov 02 13:48:30.521550 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.521456    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-util\" (UniqueName: \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-util\") pod \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\" (UID: \"8161d4b7-b679-43c2-bed4-d5db6256b9c9\") "
Nov 02 13:48:30.521951 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.521910    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-bundle" pod "8161d4b7-b679-43c2-bed4-d5db6256b9c9" (UID: "8161d4b7-b679-43c2-bed4-d5db6256b9c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:30.523702 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.523673    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8161d4b7-b679-43c2-bed4-d5db6256b9c9-kube-api-access-fm2q9" pod "8161d4b7-b679-43c2-bed4-d5db6256b9c9" (UID: "8161d4b7-b679-43c2-bed4-d5db6256b9c9"). InnerVolumeSpecName "kube-api-access-fm2q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:30.529720 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.529524    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-util" pod "8161d4b7-b679-43c2-bed4-d5db6256b9c9" (UID: "8161d4b7-b679-43c2-bed4-d5db6256b9c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:30.622614 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.622591    6284 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:30.622614 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.622611    6284 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8161d4b7-b679-43c2-bed4-d5db6256b9c9-util\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:30.622760 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:30.622620    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm2q9\" (UniqueName: \"kubernetes.io/projected/8161d4b7-b679-43c2-bed4-d5db6256b9c9-kube-api-access-fm2q9\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:31.336530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:31.336489    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk" event={"ID":"8161d4b7-b679-43c2-bed4-d5db6256b9c9","Type":"ContainerDied","Data":"b901e01e9ddcaf308cad24779ad97e4de18857f4ddf3e11a0dedf5e8abe79353"}
Nov 02 13:48:31.336947 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:31.336552    6284 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b901e01e9ddcaf308cad24779ad97e4de18857f4ddf3e11a0dedf5e8abe79353"
Nov 02 13:48:31.336947 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:31.336554    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c8dvvk"
Nov 02 13:48:34.869132 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869105    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"]
Nov 02 13:48:34.869546 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869522    6284 cpu_manager.go:414] "containerMap: removing container" podUID="1605600e-1f07-47d6-b378-6de78a500bc0" containerName="pull"
Nov 02 13:48:34.869546 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869540    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1605600e-1f07-47d6-b378-6de78a500bc0" containerName="pull"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869551    6284 cpu_manager.go:414] "containerMap: removing container" podUID="1605600e-1f07-47d6-b378-6de78a500bc0" containerName="extract"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869557    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1605600e-1f07-47d6-b378-6de78a500bc0" containerName="extract"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869569    6284 cpu_manager.go:414] "containerMap: removing container" podUID="5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" containerName="util"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869575    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" containerName="util"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869584    6284 cpu_manager.go:414] "containerMap: removing container" podUID="8161d4b7-b679-43c2-bed4-d5db6256b9c9" containerName="extract"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869591    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8161d4b7-b679-43c2-bed4-d5db6256b9c9" containerName="extract"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869607    6284 cpu_manager.go:414] "containerMap: removing container" podUID="1605600e-1f07-47d6-b378-6de78a500bc0" containerName="util"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869614    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1605600e-1f07-47d6-b378-6de78a500bc0" containerName="util"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869626    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ecd5fcd3-26b4-432b-b8d5-72155727b96c" containerName="extract"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869636    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd5fcd3-26b4-432b-b8d5-72155727b96c" containerName="extract"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869644    6284 cpu_manager.go:414] "containerMap: removing container" podUID="8161d4b7-b679-43c2-bed4-d5db6256b9c9" containerName="pull"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869648    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8161d4b7-b679-43c2-bed4-d5db6256b9c9" containerName="pull"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869655    6284 cpu_manager.go:414] "containerMap: removing container" podUID="9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" containerName="pull"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869659    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" containerName="pull"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869673    6284 cpu_manager.go:414] "containerMap: removing container" podUID="5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" containerName="pull"
Nov 02 13:48:34.869681 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869681    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" containerName="pull"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869690    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ecd5fcd3-26b4-432b-b8d5-72155727b96c" containerName="pull"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869695    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd5fcd3-26b4-432b-b8d5-72155727b96c" containerName="pull"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869701    6284 cpu_manager.go:414] "containerMap: removing container" podUID="9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869705    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869715    6284 cpu_manager.go:414] "containerMap: removing container" podUID="9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" containerName="util"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869719    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" containerName="util"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869725    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ac90c685-6134-4ccc-83c4-01f57629a4f2" containerName="util"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869729    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac90c685-6134-4ccc-83c4-01f57629a4f2" containerName="util"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869735    6284 cpu_manager.go:414] "containerMap: removing container" podUID="8161d4b7-b679-43c2-bed4-d5db6256b9c9" containerName="util"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869739    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8161d4b7-b679-43c2-bed4-d5db6256b9c9" containerName="util"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869745    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ac90c685-6134-4ccc-83c4-01f57629a4f2" containerName="pull"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869749    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac90c685-6134-4ccc-83c4-01f57629a4f2" containerName="pull"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869756    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ac90c685-6134-4ccc-83c4-01f57629a4f2" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869760    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac90c685-6134-4ccc-83c4-01f57629a4f2" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869768    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ecd5fcd3-26b4-432b-b8d5-72155727b96c" containerName="util"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869771    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd5fcd3-26b4-432b-b8d5-72155727b96c" containerName="util"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869779    6284 cpu_manager.go:414] "containerMap: removing container" podUID="5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869783    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869832    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd5fcd3-26b4-432b-b8d5-72155727b96c" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869850    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5143ef-5f0f-48e9-8e1b-6aa388d7101a" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869859    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac90c685-6134-4ccc-83c4-01f57629a4f2" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869866    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8bd1e4-e1d6-4ad4-9170-8ce900fbc4fc" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869871    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="1605600e-1f07-47d6-b378-6de78a500bc0" containerName="extract"
Nov 02 13:48:34.870205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.869876    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="8161d4b7-b679-43c2-bed4-d5db6256b9c9" containerName="extract"
Nov 02 13:48:34.888512 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.888492    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"
Nov 02 13:48:34.906006 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.905985    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"]
Nov 02 13:48:34.914365 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.914350    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:48:34.914458 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.914369    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 02 13:48:34.914458 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.914421    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-tlrnb\""
Nov 02 13:48:34.956676 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.956657    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc7vs\" (UniqueName: \"kubernetes.io/projected/08b70474-3d56-4367-a326-577a7ef0a9fa-kube-api-access-cc7vs\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr\" (UID: \"08b70474-3d56-4367-a326-577a7ef0a9fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"
Nov 02 13:48:34.956785 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:34.956699    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08b70474-3d56-4367-a326-577a7ef0a9fa-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr\" (UID: \"08b70474-3d56-4367-a326-577a7ef0a9fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"
Nov 02 13:48:35.057912 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:35.057889    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc7vs\" (UniqueName: \"kubernetes.io/projected/08b70474-3d56-4367-a326-577a7ef0a9fa-kube-api-access-cc7vs\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr\" (UID: \"08b70474-3d56-4367-a326-577a7ef0a9fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"
Nov 02 13:48:35.058034 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:35.057951    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08b70474-3d56-4367-a326-577a7ef0a9fa-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr\" (UID: \"08b70474-3d56-4367-a326-577a7ef0a9fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"
Nov 02 13:48:35.058269 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:35.058249    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08b70474-3d56-4367-a326-577a7ef0a9fa-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr\" (UID: \"08b70474-3d56-4367-a326-577a7ef0a9fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"
Nov 02 13:48:35.084039 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:35.084012    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc7vs\" (UniqueName: \"kubernetes.io/projected/08b70474-3d56-4367-a326-577a7ef0a9fa-kube-api-access-cc7vs\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr\" (UID: \"08b70474-3d56-4367-a326-577a7ef0a9fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"
Nov 02 13:48:35.214063 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:35.214007    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"
Nov 02 13:48:35.554183 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:35.554142    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr"]
Nov 02 13:48:35.558594 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:48:35.558563    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b70474_3d56_4367_a326_577a7ef0a9fa.slice/crio-eec0bd53992341f8c79b122c43547604587b3ed9f1cc3f9e764a3178171f4f88 WatchSource:0}: Error finding container eec0bd53992341f8c79b122c43547604587b3ed9f1cc3f9e764a3178171f4f88: Status 404 returned error can't find the container with id eec0bd53992341f8c79b122c43547604587b3ed9f1cc3f9e764a3178171f4f88
Nov 02 13:48:36.360582 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:36.360495    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr" event={"ID":"08b70474-3d56-4367-a326-577a7ef0a9fa","Type":"ContainerStarted","Data":"eec0bd53992341f8c79b122c43547604587b3ed9f1cc3f9e764a3178171f4f88"}
Nov 02 13:48:37.367194 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:37.367165    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr" event={"ID":"08b70474-3d56-4367-a326-577a7ef0a9fa","Type":"ContainerStarted","Data":"e36881400e36b4b764ebeea874edb5435398e79d32881fa82d930e60d55478a7"}
Nov 02 13:48:39.377957 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.377912    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 13:48:39.378929 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.378878    6284 generic.go:356] "Generic (PLEG): container finished" podID="70acb887-3146-4f03-97cc-19904033a3dc" containerID="b399d37814314458f5c247d60a41aa14e0623c1789980ed4fb22008c971826c8" exitCode=1
Nov 02 13:48:39.379085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.378954    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk" event={"ID":"70acb887-3146-4f03-97cc-19904033a3dc","Type":"ContainerDied","Data":"b399d37814314458f5c247d60a41aa14e0623c1789980ed4fb22008c971826c8"}
Nov 02 13:48:39.432403 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.432359    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-2f4cr" podStartSLOduration=4.01341738 podStartE2EDuration="5.432345656s" totalImagesPullingTime="1.418928276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:37.523440958 +0000 UTC m=+249.482816566" watchObservedRunningTime="2026-11-02 13:48:39.432345656 +0000 UTC m=+251.391721273"
Nov 02 13:48:39.433496 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.433464    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"]
Nov 02 13:48:39.436029 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.436005    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"
Nov 02 13:48:39.456626 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.456608    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 13:48:39.470622 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.470416    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"]
Nov 02 13:48:39.556635 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.556609    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 13:48:39.594115 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.594089    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4cc34b6-998c-4e72-a138-74b124ff8e79-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8bzbn\" (UID: \"c4cc34b6-998c-4e72-a138-74b124ff8e79\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"
Nov 02 13:48:39.594230 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.594129    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhnhc\" (UniqueName: \"kubernetes.io/projected/c4cc34b6-998c-4e72-a138-74b124ff8e79-kube-api-access-hhnhc\") pod \"cert-manager-webhook-5dbbbf75cc-8bzbn\" (UID: \"c4cc34b6-998c-4e72-a138-74b124ff8e79\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"
Nov 02 13:48:39.695186 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.695120    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4cc34b6-998c-4e72-a138-74b124ff8e79-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8bzbn\" (UID: \"c4cc34b6-998c-4e72-a138-74b124ff8e79\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"
Nov 02 13:48:39.695186 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.695162    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnhc\" (UniqueName: \"kubernetes.io/projected/c4cc34b6-998c-4e72-a138-74b124ff8e79-kube-api-access-hhnhc\") pod \"cert-manager-webhook-5dbbbf75cc-8bzbn\" (UID: \"c4cc34b6-998c-4e72-a138-74b124ff8e79\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"
Nov 02 13:48:39.721301 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.721270    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4cc34b6-998c-4e72-a138-74b124ff8e79-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8bzbn\" (UID: \"c4cc34b6-998c-4e72-a138-74b124ff8e79\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"
Nov 02 13:48:39.721424 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.721382    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhnhc\" (UniqueName: \"kubernetes.io/projected/c4cc34b6-998c-4e72-a138-74b124ff8e79-kube-api-access-hhnhc\") pod \"cert-manager-webhook-5dbbbf75cc-8bzbn\" (UID: \"c4cc34b6-998c-4e72-a138-74b124ff8e79\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"
Nov 02 13:48:39.757710 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:39.757688    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"
Nov 02 13:48:40.067913 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.067882    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"]
Nov 02 13:48:40.072230 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:48:40.072203    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4cc34b6_998c_4e72_a138_74b124ff8e79.slice/crio-7625994f9b2226fa3b6425f9a7df606e517e96949b087e6056083acf54d2958a WatchSource:0}: Error finding container 7625994f9b2226fa3b6425f9a7df606e517e96949b087e6056083acf54d2958a: Status 404 returned error can't find the container with id 7625994f9b2226fa3b6425f9a7df606e517e96949b087e6056083acf54d2958a
Nov 02 13:48:40.384972 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.384844    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn" event={"ID":"c4cc34b6-998c-4e72-a138-74b124ff8e79","Type":"ContainerStarted","Data":"7625994f9b2226fa3b6425f9a7df606e517e96949b087e6056083acf54d2958a"}
Nov 02 13:48:40.490546 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.490521    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 13:48:40.493194 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.493171    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:48:40.552949 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.552909    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"]
Nov 02 13:48:40.553218 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.553205    6284 cpu_manager.go:414] "containerMap: removing container" podUID="70acb887-3146-4f03-97cc-19904033a3dc" containerName="pull"
Nov 02 13:48:40.553259 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.553220    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70acb887-3146-4f03-97cc-19904033a3dc" containerName="pull"
Nov 02 13:48:40.553259 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.553249    6284 cpu_manager.go:414] "containerMap: removing container" podUID="70acb887-3146-4f03-97cc-19904033a3dc" containerName="util"
Nov 02 13:48:40.553259 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.553254    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70acb887-3146-4f03-97cc-19904033a3dc" containerName="util"
Nov 02 13:48:40.553355 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.553263    6284 cpu_manager.go:414] "containerMap: removing container" podUID="70acb887-3146-4f03-97cc-19904033a3dc" containerName="extract"
Nov 02 13:48:40.553355 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.553268    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70acb887-3146-4f03-97cc-19904033a3dc" containerName="extract"
Nov 02 13:48:40.553355 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.553316    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="70acb887-3146-4f03-97cc-19904033a3dc" containerName="extract"
Nov 02 13:48:40.555284 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.555269    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.602700 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.602680    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70acb887-3146-4f03-97cc-19904033a3dc-kube-api-access-jktvf\" (UniqueName: \"kubernetes.io/projected/70acb887-3146-4f03-97cc-19904033a3dc-kube-api-access-jktvf\") pod \"70acb887-3146-4f03-97cc-19904033a3dc\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") "
Nov 02 13:48:40.602827 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.602751    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-bundle\") pod \"70acb887-3146-4f03-97cc-19904033a3dc\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") "
Nov 02 13:48:40.602827 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.602779    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-util\" (UniqueName: \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-util\") pod \"70acb887-3146-4f03-97cc-19904033a3dc\" (UID: \"70acb887-3146-4f03-97cc-19904033a3dc\") "
Nov 02 13:48:40.603252 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.603227    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-bundle" pod "70acb887-3146-4f03-97cc-19904033a3dc" (UID: "70acb887-3146-4f03-97cc-19904033a3dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:40.604725 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.604693    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70acb887-3146-4f03-97cc-19904033a3dc-kube-api-access-jktvf" pod "70acb887-3146-4f03-97cc-19904033a3dc" (UID: "70acb887-3146-4f03-97cc-19904033a3dc"). InnerVolumeSpecName "kube-api-access-jktvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:40.608188 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.608163    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-util" pod "70acb887-3146-4f03-97cc-19904033a3dc" (UID: "70acb887-3146-4f03-97cc-19904033a3dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:40.634106 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.634069    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"]
Nov 02 13:48:40.647420 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.647354    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 13:48:40.647523 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.647503    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 13:48:40.665273 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.665254    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 13:48:40.665511 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.665497    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 13:48:40.665691 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.665679    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 13:48:40.665724 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.665708    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-drxxd\""
Nov 02 13:48:40.703722 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.703698    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/547ed086-02f0-46bd-9357-b282e063ab79-manifests\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.703820 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.703725    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/547ed086-02f0-46bd-9357-b282e063ab79-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.703820 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.703744    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lqfr\" (UniqueName: \"kubernetes.io/projected/547ed086-02f0-46bd-9357-b282e063ab79-kube-api-access-5lqfr\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.703887 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.703870    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/547ed086-02f0-46bd-9357-b282e063ab79-webhook-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.703967 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.703954    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/547ed086-02f0-46bd-9357-b282e063ab79-metrics-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.704010 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.703980    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/547ed086-02f0-46bd-9357-b282e063ab79-tmp\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.704075 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.704062    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jktvf\" (UniqueName: \"kubernetes.io/projected/70acb887-3146-4f03-97cc-19904033a3dc-kube-api-access-jktvf\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:40.704120 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.704083    6284 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:40.704120 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.704097    6284 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70acb887-3146-4f03-97cc-19904033a3dc-util\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:40.805022 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.804999    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/547ed086-02f0-46bd-9357-b282e063ab79-webhook-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.805141 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.805029    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/547ed086-02f0-46bd-9357-b282e063ab79-metrics-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.805141 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.805045    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/547ed086-02f0-46bd-9357-b282e063ab79-tmp\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.805141 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.805100    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/547ed086-02f0-46bd-9357-b282e063ab79-manifests\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.805141 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.805116    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/547ed086-02f0-46bd-9357-b282e063ab79-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.805141 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.805133    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lqfr\" (UniqueName: \"kubernetes.io/projected/547ed086-02f0-46bd-9357-b282e063ab79-kube-api-access-5lqfr\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.805594 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.805566    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/547ed086-02f0-46bd-9357-b282e063ab79-manifests\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.807426 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.807402    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/547ed086-02f0-46bd-9357-b282e063ab79-tmp\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.807774 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.807752    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/547ed086-02f0-46bd-9357-b282e063ab79-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.807874 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.807857    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/547ed086-02f0-46bd-9357-b282e063ab79-webhook-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.807921 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.807900    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/547ed086-02f0-46bd-9357-b282e063ab79-metrics-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.826759 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.826738    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lqfr\" (UniqueName: \"kubernetes.io/projected/547ed086-02f0-46bd-9357-b282e063ab79-kube-api-access-5lqfr\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-zl9nb\" (UID: \"547ed086-02f0-46bd-9357-b282e063ab79\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:40.867747 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:40.867724    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:41.257233 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:41.257202    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"]
Nov 02 13:48:41.271217 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:48:41.271181    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547ed086_02f0_46bd_9357_b282e063ab79.slice/crio-9c55b1d45f81c91dfd34ad14c0ab3d8d240d86fe163e47ab46d5bab9bf01be3f WatchSource:0}: Error finding container 9c55b1d45f81c91dfd34ad14c0ab3d8d240d86fe163e47ab46d5bab9bf01be3f: Status 404 returned error can't find the container with id 9c55b1d45f81c91dfd34ad14c0ab3d8d240d86fe163e47ab46d5bab9bf01be3f
Nov 02 13:48:41.391262 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:41.391232    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 13:48:41.392119 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:41.392089    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk" event={"ID":"70acb887-3146-4f03-97cc-19904033a3dc","Type":"ContainerDied","Data":"4c96e7a7520a5f0a7df3696362043dda82f1e3f97ddb63af95abe12b9ecd585c"}
Nov 02 13:48:41.392119 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:41.392118    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk"
Nov 02 13:48:41.392291 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:41.392168    6284 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c96e7a7520a5f0a7df3696362043dda82f1e3f97ddb63af95abe12b9ecd585c"
Nov 02 13:48:41.394001 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:41.393966    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb" event={"ID":"547ed086-02f0-46bd-9357-b282e063ab79","Type":"ContainerStarted","Data":"9c55b1d45f81c91dfd34ad14c0ab3d8d240d86fe163e47ab46d5bab9bf01be3f"}
Nov 02 13:48:42.845841 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:42.845807    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"]
Nov 02 13:48:42.852388 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:42.851864    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"
Nov 02 13:48:42.868882 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:42.868717    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"]
Nov 02 13:48:42.925293 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:42.925265    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lf4p\" (UniqueName: \"kubernetes.io/projected/72e9b683-09fd-4176-bb7b-a785bf96544a-kube-api-access-6lf4p\") pod \"cert-manager-cainjector-5c57bbd4fc-wr8sl\" (UID: \"72e9b683-09fd-4176-bb7b-a785bf96544a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"
Nov 02 13:48:42.925450 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:42.925327    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72e9b683-09fd-4176-bb7b-a785bf96544a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wr8sl\" (UID: \"72e9b683-09fd-4176-bb7b-a785bf96544a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"
Nov 02 13:48:42.958489 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:42.958462    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x5dmj\""
Nov 02 13:48:43.025975 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.025928    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lf4p\" (UniqueName: \"kubernetes.io/projected/72e9b683-09fd-4176-bb7b-a785bf96544a-kube-api-access-6lf4p\") pod \"cert-manager-cainjector-5c57bbd4fc-wr8sl\" (UID: \"72e9b683-09fd-4176-bb7b-a785bf96544a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"
Nov 02 13:48:43.026142 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.026038    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72e9b683-09fd-4176-bb7b-a785bf96544a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wr8sl\" (UID: \"72e9b683-09fd-4176-bb7b-a785bf96544a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"
Nov 02 13:48:43.052633 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.052606    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/72e9b683-09fd-4176-bb7b-a785bf96544a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wr8sl\" (UID: \"72e9b683-09fd-4176-bb7b-a785bf96544a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"
Nov 02 13:48:43.052871 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.052851    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lf4p\" (UniqueName: \"kubernetes.io/projected/72e9b683-09fd-4176-bb7b-a785bf96544a-kube-api-access-6lf4p\") pod \"cert-manager-cainjector-5c57bbd4fc-wr8sl\" (UID: \"72e9b683-09fd-4176-bb7b-a785bf96544a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"
Nov 02 13:48:43.165071 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.164983    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"
Nov 02 13:48:43.481589 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.481557    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl"]
Nov 02 13:48:43.486357 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:48:43.486333    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72e9b683_09fd_4176_bb7b_a785bf96544a.slice/crio-ce9651a05940658e7965230b51d757a467e0a731d637b5f6fbce134b14101f86 WatchSource:0}: Error finding container ce9651a05940658e7965230b51d757a467e0a731d637b5f6fbce134b14101f86: Status 404 returned error can't find the container with id ce9651a05940658e7965230b51d757a467e0a731d637b5f6fbce134b14101f86
Nov 02 13:48:43.897927 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:48:43.897893    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:48:43.945901 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.945870    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"]
Nov 02 13:48:43.948613 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.948586    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"
Nov 02 13:48:43.985969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.972681    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-ghg7z\""
Nov 02 13:48:43.985969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.972809    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 13:48:43.985969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.972840    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:48:43.985969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:43.980614    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"]
Nov 02 13:48:44.035859 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.035826    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpbk\" (UniqueName: \"kubernetes.io/projected/dac6547a-02b5-44df-a48b-d0ae873a1f44-kube-api-access-ggpbk\") pod \"jobset-operator-674cd59dc5-kjr6d\" (UID: \"dac6547a-02b5-44df-a48b-d0ae873a1f44\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"
Nov 02 13:48:44.036036 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.035888    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac6547a-02b5-44df-a48b-d0ae873a1f44-tmp\") pod \"jobset-operator-674cd59dc5-kjr6d\" (UID: \"dac6547a-02b5-44df-a48b-d0ae873a1f44\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"
Nov 02 13:48:44.137332 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.137301    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpbk\" (UniqueName: \"kubernetes.io/projected/dac6547a-02b5-44df-a48b-d0ae873a1f44-kube-api-access-ggpbk\") pod \"jobset-operator-674cd59dc5-kjr6d\" (UID: \"dac6547a-02b5-44df-a48b-d0ae873a1f44\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"
Nov 02 13:48:44.137332 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.137343    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac6547a-02b5-44df-a48b-d0ae873a1f44-tmp\") pod \"jobset-operator-674cd59dc5-kjr6d\" (UID: \"dac6547a-02b5-44df-a48b-d0ae873a1f44\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"
Nov 02 13:48:44.137717 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.137683    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dac6547a-02b5-44df-a48b-d0ae873a1f44-tmp\") pod \"jobset-operator-674cd59dc5-kjr6d\" (UID: \"dac6547a-02b5-44df-a48b-d0ae873a1f44\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"
Nov 02 13:48:44.167350 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.167283    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpbk\" (UniqueName: \"kubernetes.io/projected/dac6547a-02b5-44df-a48b-d0ae873a1f44-kube-api-access-ggpbk\") pod \"jobset-operator-674cd59dc5-kjr6d\" (UID: \"dac6547a-02b5-44df-a48b-d0ae873a1f44\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"
Nov 02 13:48:44.262320 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.262288    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"
Nov 02 13:48:44.414956 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.414905    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn" event={"ID":"c4cc34b6-998c-4e72-a138-74b124ff8e79","Type":"ContainerStarted","Data":"2743384bab60cc4a8590768ffeb3518478cd83dc83cd0dceccbed6caf2a7f8bc"}
Nov 02 13:48:44.415138 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.415113    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"
Nov 02 13:48:44.417075 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.417044    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl" event={"ID":"72e9b683-09fd-4176-bb7b-a785bf96544a","Type":"ContainerStarted","Data":"2f329460539224d0ea952f50fc3acabaf1a002b3cdffbdf8265a18f5b3323ebc"}
Nov 02 13:48:44.417205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.417087    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl" event={"ID":"72e9b683-09fd-4176-bb7b-a785bf96544a","Type":"ContainerStarted","Data":"ce9651a05940658e7965230b51d757a467e0a731d637b5f6fbce134b14101f86"}
Nov 02 13:48:44.419323 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.419260    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb" event={"ID":"547ed086-02f0-46bd-9357-b282e063ab79","Type":"ContainerStarted","Data":"c681a100b4f6381c28933b81ec6e5d371a1c483082d43d602932bbab234a8203"}
Nov 02 13:48:44.455125 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.455084    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn" podStartSLOduration=2.13689303 podStartE2EDuration="5.455071992s" totalImagesPullingTime="3.318178962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:44.454837482 +0000 UTC m=+256.414213089" watchObservedRunningTime="2026-11-02 13:48:44.455071992 +0000 UTC m=+256.414447600"
Nov 02 13:48:44.587678 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.587634    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wr8sl" podStartSLOduration=2.587620386 podStartE2EDuration="2.587620386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:44.586547415 +0000 UTC m=+256.545923037" watchObservedRunningTime="2026-11-02 13:48:44.587620386 +0000 UTC m=+256.546995993"
Nov 02 13:48:44.650156 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:44.650115    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d"]
Nov 02 13:48:44.655756 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:48:44.655713    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddac6547a_02b5_44df_a48b_d0ae873a1f44.slice/crio-847b67f430f815d3cfcb03b0502f65c1d4450dbab968c9590e3533830299dcaa WatchSource:0}: Error finding container 847b67f430f815d3cfcb03b0502f65c1d4450dbab968c9590e3533830299dcaa: Status 404 returned error can't find the container with id 847b67f430f815d3cfcb03b0502f65c1d4450dbab968c9590e3533830299dcaa
Nov 02 13:48:45.427188 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:45.427151    6284 generic.go:356] "Generic (PLEG): container finished" podID="547ed086-02f0-46bd-9357-b282e063ab79" containerID="c681a100b4f6381c28933b81ec6e5d371a1c483082d43d602932bbab234a8203" exitCode=0
Nov 02 13:48:45.427697 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:45.427255    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb" event={"ID":"547ed086-02f0-46bd-9357-b282e063ab79","Type":"ContainerDied","Data":"c681a100b4f6381c28933b81ec6e5d371a1c483082d43d602932bbab234a8203"}
Nov 02 13:48:45.429031 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:45.428861    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d" event={"ID":"dac6547a-02b5-44df-a48b-d0ae873a1f44","Type":"ContainerStarted","Data":"847b67f430f815d3cfcb03b0502f65c1d4450dbab968c9590e3533830299dcaa"}
Nov 02 13:48:46.436102 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:46.436066    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb" event={"ID":"547ed086-02f0-46bd-9357-b282e063ab79","Type":"ContainerStarted","Data":"83fa2f43877f7f0fb3bd2caf5d024960b54bea37c1dab2a76c9728ab4ab8c47e"}
Nov 02 13:48:46.436536 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:46.436168    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:48:46.437775 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:46.437753    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d" event={"ID":"dac6547a-02b5-44df-a48b-d0ae873a1f44","Type":"ContainerStarted","Data":"5071a1118af42ad1ed10461bf43ca35995e4dd7293ce292015790c2807fdadae"}
Nov 02 13:48:46.590685 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:46.590597    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb" podStartSLOduration=2.812702267 podStartE2EDuration="6.590583491s" totalImagesPullingTime="3.282926044s" totalInitContainerRuntime="494.95518ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:46.579982236 +0000 UTC m=+258.539357840" watchObservedRunningTime="2026-11-02 13:48:46.590583491 +0000 UTC m=+258.549959098"
Nov 02 13:48:48.061348 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.061295    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-kjr6d" podStartSLOduration=3.472606338 podStartE2EDuration="5.061280173s" totalImagesPullingTime="1.588673835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:46.624753883 +0000 UTC m=+258.584129482" watchObservedRunningTime="2026-11-02 13:48:48.061280173 +0000 UTC m=+260.020655779"
Nov 02 13:48:48.061960 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.061916    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"]
Nov 02 13:48:48.065446 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.065429    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.089105 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.089085    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-bjkn8\""
Nov 02 13:48:48.089190 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.089107    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 13:48:48.089190 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.089107    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 13:48:48.090658 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.090639    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"]
Nov 02 13:48:48.168580 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.168557    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.168698 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.168585    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7pnw\" (UniqueName: \"kubernetes.io/projected/17a6d8dc-c5bb-40a6-8067-245c69781194-kube-api-access-s7pnw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.168698 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.168618    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.270031 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.270003    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7pnw\" (UniqueName: \"kubernetes.io/projected/17a6d8dc-c5bb-40a6-8067-245c69781194-kube-api-access-s7pnw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.270162 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.270060    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.270162 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.270137    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.270447 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.270429    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.270486 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.270458    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.292518 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.292492    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7pnw\" (UniqueName: \"kubernetes.io/projected/17a6d8dc-c5bb-40a6-8067-245c69781194-kube-api-access-s7pnw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.380226 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.380136    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:48.687255 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:48.687167    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"]
Nov 02 13:48:48.693647 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:48:48.693615    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a6d8dc_c5bb_40a6_8067_245c69781194.slice/crio-2d9cd96f0a51dc197041c08217435578b5485c07627ea8b93f05d966bd5dacde WatchSource:0}: Error finding container 2d9cd96f0a51dc197041c08217435578b5485c07627ea8b93f05d966bd5dacde: Status 404 returned error can't find the container with id 2d9cd96f0a51dc197041c08217435578b5485c07627ea8b93f05d966bd5dacde
Nov 02 13:48:49.456407 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.456372    6284 generic.go:356] "Generic (PLEG): container finished" podID="17a6d8dc-c5bb-40a6-8067-245c69781194" containerID="851b7cba3a6e919024cfb27f9e835e3147e40b8f23aad992792f6d16f2f9a550" exitCode=0
Nov 02 13:48:49.456857 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.456411    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q" event={"ID":"17a6d8dc-c5bb-40a6-8067-245c69781194","Type":"ContainerDied","Data":"851b7cba3a6e919024cfb27f9e835e3147e40b8f23aad992792f6d16f2f9a550"}
Nov 02 13:48:49.456857 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.456472    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q" event={"ID":"17a6d8dc-c5bb-40a6-8067-245c69781194","Type":"ContainerStarted","Data":"2d9cd96f0a51dc197041c08217435578b5485c07627ea8b93f05d966bd5dacde"}
Nov 02 13:48:49.700351 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.700266    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-94hfw"]
Nov 02 13:48:49.706242 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.706224    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-94hfw"
Nov 02 13:48:49.723388 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.723365    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hpkq7\""
Nov 02 13:48:49.728826 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.728788    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-94hfw"]
Nov 02 13:48:49.785892 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.785862    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d703382-f397-4fbe-a9c3-a848d29999c9-bound-sa-token\") pod \"cert-manager-597b95cb84-94hfw\" (UID: \"8d703382-f397-4fbe-a9c3-a848d29999c9\") " pod="cert-manager/cert-manager-597b95cb84-94hfw"
Nov 02 13:48:49.786034 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.785918    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpr97\" (UniqueName: \"kubernetes.io/projected/8d703382-f397-4fbe-a9c3-a848d29999c9-kube-api-access-tpr97\") pod \"cert-manager-597b95cb84-94hfw\" (UID: \"8d703382-f397-4fbe-a9c3-a848d29999c9\") " pod="cert-manager/cert-manager-597b95cb84-94hfw"
Nov 02 13:48:49.887157 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.887128    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d703382-f397-4fbe-a9c3-a848d29999c9-bound-sa-token\") pod \"cert-manager-597b95cb84-94hfw\" (UID: \"8d703382-f397-4fbe-a9c3-a848d29999c9\") " pod="cert-manager/cert-manager-597b95cb84-94hfw"
Nov 02 13:48:49.887281 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.887182    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpr97\" (UniqueName: \"kubernetes.io/projected/8d703382-f397-4fbe-a9c3-a848d29999c9-kube-api-access-tpr97\") pod \"cert-manager-597b95cb84-94hfw\" (UID: \"8d703382-f397-4fbe-a9c3-a848d29999c9\") " pod="cert-manager/cert-manager-597b95cb84-94hfw"
Nov 02 13:48:49.913058 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.913030    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d703382-f397-4fbe-a9c3-a848d29999c9-bound-sa-token\") pod \"cert-manager-597b95cb84-94hfw\" (UID: \"8d703382-f397-4fbe-a9c3-a848d29999c9\") " pod="cert-manager/cert-manager-597b95cb84-94hfw"
Nov 02 13:48:49.913183 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:49.913167    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpr97\" (UniqueName: \"kubernetes.io/projected/8d703382-f397-4fbe-a9c3-a848d29999c9-kube-api-access-tpr97\") pod \"cert-manager-597b95cb84-94hfw\" (UID: \"8d703382-f397-4fbe-a9c3-a848d29999c9\") " pod="cert-manager/cert-manager-597b95cb84-94hfw"
Nov 02 13:48:50.018137 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:50.018114    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-94hfw"
Nov 02 13:48:50.321904 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:50.321818    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-94hfw"]
Nov 02 13:48:50.325672 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:48:50.325644    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d703382_f397_4fbe_a9c3_a848d29999c9.slice/crio-7e073c45101019c8d9ea1607ff53575a8f6742968025b23c388f84d50e26503e WatchSource:0}: Error finding container 7e073c45101019c8d9ea1607ff53575a8f6742968025b23c388f84d50e26503e: Status 404 returned error can't find the container with id 7e073c45101019c8d9ea1607ff53575a8f6742968025b23c388f84d50e26503e
Nov 02 13:48:50.431088 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:50.431055    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8bzbn"
Nov 02 13:48:50.462838 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:50.462805    6284 generic.go:356] "Generic (PLEG): container finished" podID="17a6d8dc-c5bb-40a6-8067-245c69781194" containerID="d2605550bdfe337c62054ddda9374dd1cb74008e50e1f74823e95a0974a74880" exitCode=0
Nov 02 13:48:50.463309 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:50.462891    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q" event={"ID":"17a6d8dc-c5bb-40a6-8067-245c69781194","Type":"ContainerDied","Data":"d2605550bdfe337c62054ddda9374dd1cb74008e50e1f74823e95a0974a74880"}
Nov 02 13:48:50.464711 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:50.464693    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-94hfw" event={"ID":"8d703382-f397-4fbe-a9c3-a848d29999c9","Type":"ContainerStarted","Data":"0fee9d73d7a4432086ce8684926500f4a328cf116f5992abb6c1f7c022f0378f"}
Nov 02 13:48:50.464813 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:50.464721    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-94hfw" event={"ID":"8d703382-f397-4fbe-a9c3-a848d29999c9","Type":"ContainerStarted","Data":"7e073c45101019c8d9ea1607ff53575a8f6742968025b23c388f84d50e26503e"}
Nov 02 13:48:50.558237 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:50.558176    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-94hfw" podStartSLOduration=1.5581585759999999 podStartE2EDuration="1.558158576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:50.557657068 +0000 UTC m=+262.517032675" watchObservedRunningTime="2026-11-02 13:48:50.558158576 +0000 UTC m=+262.517534185"
Nov 02 13:48:51.471954 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:51.471908    6284 generic.go:356] "Generic (PLEG): container finished" podID="17a6d8dc-c5bb-40a6-8067-245c69781194" containerID="03652d5d818701dce179b844d68db56db034f429728b507c6f6da811ef7a400f" exitCode=0
Nov 02 13:48:51.472349 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:51.471971    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q" event={"ID":"17a6d8dc-c5bb-40a6-8067-245c69781194","Type":"ContainerDied","Data":"03652d5d818701dce179b844d68db56db034f429728b507c6f6da811ef7a400f"}
Nov 02 13:48:52.600194 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:52.600169    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:52.709140 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:52.709104    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a6d8dc-c5bb-40a6-8067-245c69781194-kube-api-access-s7pnw\" (UniqueName: \"kubernetes.io/projected/17a6d8dc-c5bb-40a6-8067-245c69781194-kube-api-access-s7pnw\") pod \"17a6d8dc-c5bb-40a6-8067-245c69781194\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") "
Nov 02 13:48:52.709276 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:52.709196    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-util\" (UniqueName: \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-util\") pod \"17a6d8dc-c5bb-40a6-8067-245c69781194\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") "
Nov 02 13:48:52.709319 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:52.709274    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-bundle\" (UniqueName: \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-bundle\") pod \"17a6d8dc-c5bb-40a6-8067-245c69781194\" (UID: \"17a6d8dc-c5bb-40a6-8067-245c69781194\") "
Nov 02 13:48:52.709774 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:52.709743    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-bundle" pod "17a6d8dc-c5bb-40a6-8067-245c69781194" (UID: "17a6d8dc-c5bb-40a6-8067-245c69781194"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:52.711179 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:52.711155    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a6d8dc-c5bb-40a6-8067-245c69781194-kube-api-access-s7pnw" pod "17a6d8dc-c5bb-40a6-8067-245c69781194" (UID: "17a6d8dc-c5bb-40a6-8067-245c69781194"). InnerVolumeSpecName "kube-api-access-s7pnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:48:52.716024 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:52.716001    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-util" pod "17a6d8dc-c5bb-40a6-8067-245c69781194" (UID: "17a6d8dc-c5bb-40a6-8067-245c69781194"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:48:52.810171 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:52.810149    6284 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-util\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:52.810171 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:52.810168    6284 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17a6d8dc-c5bb-40a6-8067-245c69781194-bundle\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:52.810302 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:52.810176    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7pnw\" (UniqueName: \"kubernetes.io/projected/17a6d8dc-c5bb-40a6-8067-245c69781194-kube-api-access-s7pnw\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:48:53.483056 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:53.483022    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q" event={"ID":"17a6d8dc-c5bb-40a6-8067-245c69781194","Type":"ContainerDied","Data":"2d9cd96f0a51dc197041c08217435578b5485c07627ea8b93f05d966bd5dacde"}
Nov 02 13:48:53.483227 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:53.483070    6284 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d9cd96f0a51dc197041c08217435578b5485c07627ea8b93f05d966bd5dacde"
Nov 02 13:48:53.483227 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:53.483035    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76z68q"
Nov 02 13:48:57.445441 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:48:57.445410    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-zl9nb"
Nov 02 13:49:12.499371 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.499328    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"]
Nov 02 13:49:12.499874 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.499799    6284 cpu_manager.go:414] "containerMap: removing container" podUID="17a6d8dc-c5bb-40a6-8067-245c69781194" containerName="util"
Nov 02 13:49:12.499874 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.499817    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a6d8dc-c5bb-40a6-8067-245c69781194" containerName="util"
Nov 02 13:49:12.499874 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.499835    6284 cpu_manager.go:414] "containerMap: removing container" podUID="17a6d8dc-c5bb-40a6-8067-245c69781194" containerName="extract"
Nov 02 13:49:12.499874 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.499842    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a6d8dc-c5bb-40a6-8067-245c69781194" containerName="extract"
Nov 02 13:49:12.499874 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.499873    6284 cpu_manager.go:414] "containerMap: removing container" podUID="17a6d8dc-c5bb-40a6-8067-245c69781194" containerName="pull"
Nov 02 13:49:12.500184 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.499880    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a6d8dc-c5bb-40a6-8067-245c69781194" containerName="pull"
Nov 02 13:49:12.500184 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.499977    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a6d8dc-c5bb-40a6-8067-245c69781194" containerName="extract"
Nov 02 13:49:12.505733 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.505712    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:12.531740 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.531717    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t7pr2\""
Nov 02 13:49:12.531872 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.531753    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 13:49:12.531872 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.531758    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 13:49:12.531872 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.531785    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 13:49:12.531872 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.531807    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 13:49:12.534187 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.534167    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"]
Nov 02 13:49:12.570442 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.570417    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d440b792-e072-43f3-9569-50a6b20f6070-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zjqv8\" (UID: \"d440b792-e072-43f3-9569-50a6b20f6070\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:12.570577 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.570455    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d440b792-e072-43f3-9569-50a6b20f6070-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zjqv8\" (UID: \"d440b792-e072-43f3-9569-50a6b20f6070\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:12.570643 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.570594    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdld7\" (UniqueName: \"kubernetes.io/projected/d440b792-e072-43f3-9569-50a6b20f6070-kube-api-access-jdld7\") pod \"kuadrant-console-plugin-66c4fb575d-zjqv8\" (UID: \"d440b792-e072-43f3-9569-50a6b20f6070\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:12.671012 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.670977    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdld7\" (UniqueName: \"kubernetes.io/projected/d440b792-e072-43f3-9569-50a6b20f6070-kube-api-access-jdld7\") pod \"kuadrant-console-plugin-66c4fb575d-zjqv8\" (UID: \"d440b792-e072-43f3-9569-50a6b20f6070\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:12.671207 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.671038    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d440b792-e072-43f3-9569-50a6b20f6070-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zjqv8\" (UID: \"d440b792-e072-43f3-9569-50a6b20f6070\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:12.671207 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.671055    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d440b792-e072-43f3-9569-50a6b20f6070-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zjqv8\" (UID: \"d440b792-e072-43f3-9569-50a6b20f6070\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:12.671635 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.671615    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d440b792-e072-43f3-9569-50a6b20f6070-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zjqv8\" (UID: \"d440b792-e072-43f3-9569-50a6b20f6070\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:12.673381 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.673357    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d440b792-e072-43f3-9569-50a6b20f6070-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zjqv8\" (UID: \"d440b792-e072-43f3-9569-50a6b20f6070\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:12.696224 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.696193    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdld7\" (UniqueName: \"kubernetes.io/projected/d440b792-e072-43f3-9569-50a6b20f6070-kube-api-access-jdld7\") pod \"kuadrant-console-plugin-66c4fb575d-zjqv8\" (UID: \"d440b792-e072-43f3-9569-50a6b20f6070\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:12.822562 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:12.822495    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"
Nov 02 13:49:13.119247 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:13.119144    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8"]
Nov 02 13:49:13.122999 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:49:13.122972    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd440b792_e072_43f3_9569_50a6b20f6070.slice/crio-610453d859ab00aadbe5260ccf28e3c8af53d2fcfc382019a3bc5fae735b5cd9 WatchSource:0}: Error finding container 610453d859ab00aadbe5260ccf28e3c8af53d2fcfc382019a3bc5fae735b5cd9: Status 404 returned error can't find the container with id 610453d859ab00aadbe5260ccf28e3c8af53d2fcfc382019a3bc5fae735b5cd9
Nov 02 13:49:13.574331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:13.574299    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8" event={"ID":"d440b792-e072-43f3-9569-50a6b20f6070","Type":"ContainerStarted","Data":"610453d859ab00aadbe5260ccf28e3c8af53d2fcfc382019a3bc5fae735b5cd9"}
Nov 02 13:49:13.909660 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:49:13.909557    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:49:18.602357 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:18.602320    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8" event={"ID":"d440b792-e072-43f3-9569-50a6b20f6070","Type":"ContainerStarted","Data":"a0ae6540aecc44c90fe5610a31e758ac764268a30fbf8e2b6331c6b69ab38464"}
Nov 02 13:49:18.651484 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:18.651436    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zjqv8" podStartSLOduration=1.844425589 podStartE2EDuration="6.651423197s" totalImagesPullingTime="4.806997608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:18.650925145 +0000 UTC m=+290.610300752" watchObservedRunningTime="2026-11-02 13:49:18.651423197 +0000 UTC m=+290.610798804"
Nov 02 13:49:26.734125 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.734089    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"]
Nov 02 13:49:26.737773 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.737756    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:26.752240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.752219    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 13:49:26.761968 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.761944    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 13:49:26.766162 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.766140    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 13:49:26.766481 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.766460    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4tb9q\""
Nov 02 13:49:26.768677 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.768654    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"]
Nov 02 13:49:26.788111 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.788086    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3827235-f5e4-44cb-8b10-af6b6d82b14b-tls-certs\") pod \"kube-auth-proxy-6fd64f76dd-lv2hm\" (UID: \"c3827235-f5e4-44cb-8b10-af6b6d82b14b\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:26.788229 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.788137    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmjd9\" (UniqueName: \"kubernetes.io/projected/c3827235-f5e4-44cb-8b10-af6b6d82b14b-kube-api-access-dmjd9\") pod \"kube-auth-proxy-6fd64f76dd-lv2hm\" (UID: \"c3827235-f5e4-44cb-8b10-af6b6d82b14b\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:26.788292 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.788273    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3827235-f5e4-44cb-8b10-af6b6d82b14b-tmp\") pod \"kube-auth-proxy-6fd64f76dd-lv2hm\" (UID: \"c3827235-f5e4-44cb-8b10-af6b6d82b14b\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:26.855870 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.855835    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 13:49:26.888865 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.888833    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3827235-f5e4-44cb-8b10-af6b6d82b14b-tls-certs\") pod \"kube-auth-proxy-6fd64f76dd-lv2hm\" (UID: \"c3827235-f5e4-44cb-8b10-af6b6d82b14b\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:26.889006 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.888871    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmjd9\" (UniqueName: \"kubernetes.io/projected/c3827235-f5e4-44cb-8b10-af6b6d82b14b-kube-api-access-dmjd9\") pod \"kube-auth-proxy-6fd64f76dd-lv2hm\" (UID: \"c3827235-f5e4-44cb-8b10-af6b6d82b14b\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:26.889006 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.888903    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3827235-f5e4-44cb-8b10-af6b6d82b14b-tmp\") pod \"kube-auth-proxy-6fd64f76dd-lv2hm\" (UID: \"c3827235-f5e4-44cb-8b10-af6b6d82b14b\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:26.891156 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.891131    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3827235-f5e4-44cb-8b10-af6b6d82b14b-tmp\") pod \"kube-auth-proxy-6fd64f76dd-lv2hm\" (UID: \"c3827235-f5e4-44cb-8b10-af6b6d82b14b\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:26.891291 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.891272    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3827235-f5e4-44cb-8b10-af6b6d82b14b-tls-certs\") pod \"kube-auth-proxy-6fd64f76dd-lv2hm\" (UID: \"c3827235-f5e4-44cb-8b10-af6b6d82b14b\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:26.909396 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:26.909375    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmjd9\" (UniqueName: \"kubernetes.io/projected/c3827235-f5e4-44cb-8b10-af6b6d82b14b-kube-api-access-dmjd9\") pod \"kube-auth-proxy-6fd64f76dd-lv2hm\" (UID: \"c3827235-f5e4-44cb-8b10-af6b6d82b14b\") " pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:27.050608 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:27.050537    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"
Nov 02 13:49:27.369968 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:27.368601    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm"]
Nov 02 13:49:27.651380 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:27.651297    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm" event={"ID":"c3827235-f5e4-44cb-8b10-af6b6d82b14b","Type":"ContainerStarted","Data":"4c8f899eb7a01a13dda6c43690c130d00dc943b8484bb98b364d6d7f4a75a9f1"}
Nov 02 13:49:27.882608 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:27.882569    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nms5l"]
Nov 02 13:49:27.887205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:27.887188    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:27.923148 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:27.923084    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nms5l"]
Nov 02 13:49:27.970967 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:27.970918    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 13:49:27.997987 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:27.997632    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mkk\" (UniqueName: \"kubernetes.io/projected/b9954cc4-cdd8-47db-8d66-7c0fe628587c-kube-api-access-s5mkk\") pod \"limitador-limitador-74dfcfb894-nms5l\" (UID: \"b9954cc4-cdd8-47db-8d66-7c0fe628587c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:27.997987 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:27.997715    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9954cc4-cdd8-47db-8d66-7c0fe628587c-config-file\") pod \"limitador-limitador-74dfcfb894-nms5l\" (UID: \"b9954cc4-cdd8-47db-8d66-7c0fe628587c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:28.039139 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:28.039110    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nms5l"]
Nov 02 13:49:28.098449 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:28.098412    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mkk\" (UniqueName: \"kubernetes.io/projected/b9954cc4-cdd8-47db-8d66-7c0fe628587c-kube-api-access-s5mkk\") pod \"limitador-limitador-74dfcfb894-nms5l\" (UID: \"b9954cc4-cdd8-47db-8d66-7c0fe628587c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:28.098613 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:28.098491    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9954cc4-cdd8-47db-8d66-7c0fe628587c-config-file\") pod \"limitador-limitador-74dfcfb894-nms5l\" (UID: \"b9954cc4-cdd8-47db-8d66-7c0fe628587c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:28.099350 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:28.099276    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9954cc4-cdd8-47db-8d66-7c0fe628587c-config-file\") pod \"limitador-limitador-74dfcfb894-nms5l\" (UID: \"b9954cc4-cdd8-47db-8d66-7c0fe628587c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:28.125281 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:28.125251    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mkk\" (UniqueName: \"kubernetes.io/projected/b9954cc4-cdd8-47db-8d66-7c0fe628587c-kube-api-access-s5mkk\") pod \"limitador-limitador-74dfcfb894-nms5l\" (UID: \"b9954cc4-cdd8-47db-8d66-7c0fe628587c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:28.206883 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:28.206817    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:28.215774 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:28.215693    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 13:49:28.216772 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:28.216692    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 13:49:28.537664 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:28.537633    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nms5l"]
Nov 02 13:49:28.541285 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:49:28.541248    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9954cc4_cdd8_47db_8d66_7c0fe628587c.slice/crio-20b09d6044177725c4d420435cd1e6286483bda6f71de90a3da1f1859f532790 WatchSource:0}: Error finding container 20b09d6044177725c4d420435cd1e6286483bda6f71de90a3da1f1859f532790: Status 404 returned error can't find the container with id 20b09d6044177725c4d420435cd1e6286483bda6f71de90a3da1f1859f532790
Nov 02 13:49:28.657827 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:28.657792    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l" event={"ID":"b9954cc4-cdd8-47db-8d66-7c0fe628587c","Type":"ContainerStarted","Data":"20b09d6044177725c4d420435cd1e6286483bda6f71de90a3da1f1859f532790"}
Nov 02 13:49:29.497898 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:29.497860    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6lnnf"]
Nov 02 13:49:29.505091 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:29.505016    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf"
Nov 02 13:49:29.521392 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:29.521370    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6lnnf"]
Nov 02 13:49:29.534060 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:29.534021    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-c5j78\""
Nov 02 13:49:29.610863 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:29.610826    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j29qb\" (UniqueName: \"kubernetes.io/projected/9fe2507d-e0d9-4097-8d86-a5836a5be1be-kube-api-access-j29qb\") pod \"authorino-56c6c9ddd5-6lnnf\" (UID: \"9fe2507d-e0d9-4097-8d86-a5836a5be1be\") " pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf"
Nov 02 13:49:29.711470 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:29.711431    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j29qb\" (UniqueName: \"kubernetes.io/projected/9fe2507d-e0d9-4097-8d86-a5836a5be1be-kube-api-access-j29qb\") pod \"authorino-56c6c9ddd5-6lnnf\" (UID: \"9fe2507d-e0d9-4097-8d86-a5836a5be1be\") " pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf"
Nov 02 13:49:29.733513 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:29.733484    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j29qb\" (UniqueName: \"kubernetes.io/projected/9fe2507d-e0d9-4097-8d86-a5836a5be1be-kube-api-access-j29qb\") pod \"authorino-56c6c9ddd5-6lnnf\" (UID: \"9fe2507d-e0d9-4097-8d86-a5836a5be1be\") " pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf"
Nov 02 13:49:29.821502 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:29.821428    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf"
Nov 02 13:49:30.140846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:30.140821    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 13:49:30.161169 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:30.161143    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6lnnf"]
Nov 02 13:49:30.169292 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:49:30.169266    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fe2507d_e0d9_4097_8d86_a5836a5be1be.slice/crio-41dc543e40c92af1f5ab30b36de6cc4c19517bccdbf76dcdf41879fe6028553a WatchSource:0}: Error finding container 41dc543e40c92af1f5ab30b36de6cc4c19517bccdbf76dcdf41879fe6028553a: Status 404 returned error can't find the container with id 41dc543e40c92af1f5ab30b36de6cc4c19517bccdbf76dcdf41879fe6028553a
Nov 02 13:49:30.679268 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:30.679232    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm" event={"ID":"c3827235-f5e4-44cb-8b10-af6b6d82b14b","Type":"ContainerStarted","Data":"fc35467da729254e851290c4f185874d67174c9fc57a07dc50e86cd18d72701e"}
Nov 02 13:49:30.681262 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:30.681209    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf" event={"ID":"9fe2507d-e0d9-4097-8d86-a5836a5be1be","Type":"ContainerStarted","Data":"41dc543e40c92af1f5ab30b36de6cc4c19517bccdbf76dcdf41879fe6028553a"}
Nov 02 13:49:30.725690 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:30.725637    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6fd64f76dd-lv2hm" podStartSLOduration=1.972977964 podStartE2EDuration="4.725621734s" totalImagesPullingTime="2.75264377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:30.721559789 +0000 UTC m=+302.680935408" watchObservedRunningTime="2026-11-02 13:49:30.725621734 +0000 UTC m=+302.684997340"
Nov 02 13:49:31.458710 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.458677    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"]
Nov 02 13:49:31.462338 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.462318    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.478261 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.478238    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 13:49:31.478261 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.478243    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 13:49:31.487999 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.487965    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-8d4wk\""
Nov 02 13:49:31.498584 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.498563    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 13:49:31.500240 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.500219    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"]
Nov 02 13:49:31.529428 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.529400    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbbz\" (UniqueName: \"kubernetes.io/projected/9926c06f-cb52-4ff4-92fe-468bafff10bb-kube-api-access-tcbbz\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.529533 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.529446    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9926c06f-cb52-4ff4-92fe-468bafff10bb-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.529533 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.529470    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9926c06f-cb52-4ff4-92fe-468bafff10bb-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.529660 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.529567    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9926c06f-cb52-4ff4-92fe-468bafff10bb-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.630776 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.630739    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbbz\" (UniqueName: \"kubernetes.io/projected/9926c06f-cb52-4ff4-92fe-468bafff10bb-kube-api-access-tcbbz\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.630986 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.630809    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9926c06f-cb52-4ff4-92fe-468bafff10bb-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.630986 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.630842    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9926c06f-cb52-4ff4-92fe-468bafff10bb-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.630986 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.630880    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9926c06f-cb52-4ff4-92fe-468bafff10bb-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.631327 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.631292    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9926c06f-cb52-4ff4-92fe-468bafff10bb-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.631327 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.631316    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9926c06f-cb52-4ff4-92fe-468bafff10bb-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.632043 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.632019    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9926c06f-cb52-4ff4-92fe-468bafff10bb-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.657329 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.657303    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbbz\" (UniqueName: \"kubernetes.io/projected/9926c06f-cb52-4ff4-92fe-468bafff10bb-kube-api-access-tcbbz\") pod \"kserve-module-controller-manager-7566cf5fc8-dpkzz\" (UID: \"9926c06f-cb52-4ff4-92fe-468bafff10bb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:31.689071 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.689041    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l" event={"ID":"b9954cc4-cdd8-47db-8d66-7c0fe628587c","Type":"ContainerStarted","Data":"8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3"}
Nov 02 13:49:31.689470 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.689155    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:31.731323 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.731246    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l" podStartSLOduration=2.300541279 podStartE2EDuration="4.731232186s" totalImagesPullingTime="2.430690907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:31.729180073 +0000 UTC m=+303.688555681" watchObservedRunningTime="2026-11-02 13:49:31.731232186 +0000 UTC m=+303.690607794"
Nov 02 13:49:31.776686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:31.776660    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:32.114860 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:32.114832    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"]
Nov 02 13:49:32.118902 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:49:32.118873    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9926c06f_cb52_4ff4_92fe_468bafff10bb.slice/crio-6cc97def6622461c96a18da9b5ef1519b1fc57013a500db91e59d78a6b844357 WatchSource:0}: Error finding container 6cc97def6622461c96a18da9b5ef1519b1fc57013a500db91e59d78a6b844357: Status 404 returned error can't find the container with id 6cc97def6622461c96a18da9b5ef1519b1fc57013a500db91e59d78a6b844357
Nov 02 13:49:32.696287 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:32.696245    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf" event={"ID":"9fe2507d-e0d9-4097-8d86-a5836a5be1be","Type":"ContainerStarted","Data":"f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067"}
Nov 02 13:49:32.697709 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:32.697682    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz" event={"ID":"9926c06f-cb52-4ff4-92fe-468bafff10bb","Type":"ContainerStarted","Data":"6cc97def6622461c96a18da9b5ef1519b1fc57013a500db91e59d78a6b844357"}
Nov 02 13:49:32.734723 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:32.734675    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf" podStartSLOduration=1.988948666 podStartE2EDuration="3.734661537s" totalImagesPullingTime="1.745712871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:32.732067577 +0000 UTC m=+304.691443181" watchObservedRunningTime="2026-11-02 13:49:32.734661537 +0000 UTC m=+304.694037134"
Nov 02 13:49:34.708686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:34.708652    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz" event={"ID":"9926c06f-cb52-4ff4-92fe-468bafff10bb","Type":"ContainerStarted","Data":"03267ffa052e80bb9bb7c172dfa09cb192ba62651e57ef2d383c9606c32fcc09"}
Nov 02 13:49:34.709103 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:34.708722    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:34.759569 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:34.759516    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz" podStartSLOduration=2.171989821 podStartE2EDuration="3.759501245s" totalImagesPullingTime="1.587511424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:34.758173335 +0000 UTC m=+306.717548945" watchObservedRunningTime="2026-11-02 13:49:34.759501245 +0000 UTC m=+306.718876852"
Nov 02 13:49:42.657610 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:42.657553    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nms5l"]
Nov 02 13:49:42.658167 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:42.657820    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l" podUID="b9954cc4-cdd8-47db-8d66-7c0fe628587c" containerName="limitador" containerID="cri-o://8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3" gracePeriod=30
Nov 02 13:49:42.658634 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:42.658615    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:43.210108 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.210074    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:43.341052 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.341014    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9954cc4-cdd8-47db-8d66-7c0fe628587c-config-file\" (UniqueName: \"kubernetes.io/configmap/b9954cc4-cdd8-47db-8d66-7c0fe628587c-config-file\") pod \"b9954cc4-cdd8-47db-8d66-7c0fe628587c\" (UID: \"b9954cc4-cdd8-47db-8d66-7c0fe628587c\") "
Nov 02 13:49:43.341218 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.341075    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9954cc4-cdd8-47db-8d66-7c0fe628587c-kube-api-access-s5mkk\" (UniqueName: \"kubernetes.io/projected/b9954cc4-cdd8-47db-8d66-7c0fe628587c-kube-api-access-s5mkk\") pod \"b9954cc4-cdd8-47db-8d66-7c0fe628587c\" (UID: \"b9954cc4-cdd8-47db-8d66-7c0fe628587c\") "
Nov 02 13:49:43.341436 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.341410    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9954cc4-cdd8-47db-8d66-7c0fe628587c-config-file" pod "b9954cc4-cdd8-47db-8d66-7c0fe628587c" (UID: "b9954cc4-cdd8-47db-8d66-7c0fe628587c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:49:43.343436 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.343400    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9954cc4-cdd8-47db-8d66-7c0fe628587c-kube-api-access-s5mkk" pod "b9954cc4-cdd8-47db-8d66-7c0fe628587c" (UID: "b9954cc4-cdd8-47db-8d66-7c0fe628587c"). InnerVolumeSpecName "kube-api-access-s5mkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:49:43.441687 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.441644    6284 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9954cc4-cdd8-47db-8d66-7c0fe628587c-config-file\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:49:43.441687 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.441679    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5mkk\" (UniqueName: \"kubernetes.io/projected/b9954cc4-cdd8-47db-8d66-7c0fe628587c-kube-api-access-s5mkk\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:49:43.753258 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.753224    6284 generic.go:356] "Generic (PLEG): container finished" podID="b9954cc4-cdd8-47db-8d66-7c0fe628587c" containerID="8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3" exitCode=0
Nov 02 13:49:43.753691 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.753283    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l"
Nov 02 13:49:43.753691 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.753313    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l" event={"ID":"b9954cc4-cdd8-47db-8d66-7c0fe628587c","Type":"ContainerDied","Data":"8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3"}
Nov 02 13:49:43.753691 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.753376    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-nms5l" event={"ID":"b9954cc4-cdd8-47db-8d66-7c0fe628587c","Type":"ContainerDied","Data":"20b09d6044177725c4d420435cd1e6286483bda6f71de90a3da1f1859f532790"}
Nov 02 13:49:43.753691 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.753407    6284 scope.go:122] "RemoveContainer" containerID="8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3"
Nov 02 13:49:43.762693 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.762671    6284 scope.go:122] "RemoveContainer" containerID="8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3"
Nov 02 13:49:43.763067 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:49:43.763044    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3\": container with ID starting with 8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3 not found: ID does not exist" containerID="8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3"
Nov 02 13:49:43.763145 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.763076    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3"} err="failed to get container status \"8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3\": rpc error: code = NotFound desc = could not find container \"8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3\": container with ID starting with 8b2ad2d40f2e143e453994e89b1deab548a6ed233e561b0138856a25905ba7e3 not found: ID does not exist"
Nov 02 13:49:43.799379 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.799335    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nms5l"]
Nov 02 13:49:43.809402 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:43.809367    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-nms5l"]
Nov 02 13:49:43.894316 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:49:43.894273    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:49:44.168398 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:44.168303    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9954cc4-cdd8-47db-8d66-7c0fe628587c" path="/var/lib/kubelet/pods/b9954cc4-cdd8-47db-8d66-7c0fe628587c/volumes"
Nov 02 13:49:49.230908 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.230864    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"]
Nov 02 13:49:49.231387 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.231268    6284 cpu_manager.go:414] "containerMap: removing container" podUID="b9954cc4-cdd8-47db-8d66-7c0fe628587c" containerName="limitador"
Nov 02 13:49:49.231387 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.231281    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9954cc4-cdd8-47db-8d66-7c0fe628587c" containerName="limitador"
Nov 02 13:49:49.231387 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.231346    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9954cc4-cdd8-47db-8d66-7c0fe628587c" containerName="limitador"
Nov 02 13:49:49.338335 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.338293    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"]
Nov 02 13:49:49.338335 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.338329    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"
Nov 02 13:49:49.351191 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.351158    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:49:49.351788 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.351771    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:49:49.361008 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.360982    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 02 13:49:49.361227 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.361209    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jcrnk\""
Nov 02 13:49:49.500870 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.500836    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7d03cb08-5ee6-4fb7-b643-53c7b0477bfc-data\") pod \"postgres-7cdbf6dc44-nfrh8\" (UID: \"7d03cb08-5ee6-4fb7-b643-53c7b0477bfc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"
Nov 02 13:49:49.501096 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.501004    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqj5j\" (UniqueName: \"kubernetes.io/projected/7d03cb08-5ee6-4fb7-b643-53c7b0477bfc-kube-api-access-tqj5j\") pod \"postgres-7cdbf6dc44-nfrh8\" (UID: \"7d03cb08-5ee6-4fb7-b643-53c7b0477bfc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"
Nov 02 13:49:49.601530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.601482    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqj5j\" (UniqueName: \"kubernetes.io/projected/7d03cb08-5ee6-4fb7-b643-53c7b0477bfc-kube-api-access-tqj5j\") pod \"postgres-7cdbf6dc44-nfrh8\" (UID: \"7d03cb08-5ee6-4fb7-b643-53c7b0477bfc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"
Nov 02 13:49:49.601747 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.601542    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7d03cb08-5ee6-4fb7-b643-53c7b0477bfc-data\") pod \"postgres-7cdbf6dc44-nfrh8\" (UID: \"7d03cb08-5ee6-4fb7-b643-53c7b0477bfc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"
Nov 02 13:49:49.602016 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.601997    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7d03cb08-5ee6-4fb7-b643-53c7b0477bfc-data\") pod \"postgres-7cdbf6dc44-nfrh8\" (UID: \"7d03cb08-5ee6-4fb7-b643-53c7b0477bfc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"
Nov 02 13:49:49.622773 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.622737    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqj5j\" (UniqueName: \"kubernetes.io/projected/7d03cb08-5ee6-4fb7-b643-53c7b0477bfc-kube-api-access-tqj5j\") pod \"postgres-7cdbf6dc44-nfrh8\" (UID: \"7d03cb08-5ee6-4fb7-b643-53c7b0477bfc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"
Nov 02 13:49:49.666390 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:49.666353    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"
Nov 02 13:49:50.014600 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:50.014553    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"]
Nov 02 13:49:50.019194 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:49:50.019129    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d03cb08_5ee6_4fb7_b643_53c7b0477bfc.slice/crio-e694999aa09bb7d2c1cf85bd0db6c7bcc6fe29d96073c5737d1973f995e92ab1 WatchSource:0}: Error finding container e694999aa09bb7d2c1cf85bd0db6c7bcc6fe29d96073c5737d1973f995e92ab1: Status 404 returned error can't find the container with id e694999aa09bb7d2c1cf85bd0db6c7bcc6fe29d96073c5737d1973f995e92ab1
Nov 02 13:49:50.791289 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:50.791250    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8" event={"ID":"7d03cb08-5ee6-4fb7-b643-53c7b0477bfc","Type":"ContainerStarted","Data":"e694999aa09bb7d2c1cf85bd0db6c7bcc6fe29d96073c5737d1973f995e92ab1"}
Nov 02 13:49:55.715379 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:55.715346    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-dpkzz"
Nov 02 13:49:55.820131 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:55.820096    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8" event={"ID":"7d03cb08-5ee6-4fb7-b643-53c7b0477bfc","Type":"ContainerStarted","Data":"def1cd1c20388df642d7cb5ae76ea2401ebd663c7687bc516e05f4581e0ae5d6"}
Nov 02 13:49:55.820320 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:55.820213    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"
Nov 02 13:49:55.857766 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:49:55.857698    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8" podStartSLOduration=1.913964228 podStartE2EDuration="6.85767744s" totalImagesPullingTime="4.943713212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:55.855617377 +0000 UTC m=+327.814993000" watchObservedRunningTime="2026-11-02 13:49:55.85767744 +0000 UTC m=+327.817053048"
Nov 02 13:50:01.855597 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:01.855566    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nfrh8"
Nov 02 13:50:13.893692 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:50:13.893646    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:50:19.971363 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:19.971323    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-kg8n2"]
Nov 02 13:50:19.983603 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:19.983570    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:19.998646 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:19.998606    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-kg8n2"]
Nov 02 13:50:20.006253 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.006219    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 13:50:20.006447 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.006408    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jtrps\""
Nov 02 13:50:20.006447 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.006408    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 13:50:20.006544 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.006517    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 13:50:20.083797 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.083754    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-metrics-certs\") pod \"maas-controller-7f746cf4d4-kg8n2\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") " pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:20.084041 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.083821    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-webhook-cert\") pod \"maas-controller-7f746cf4d4-kg8n2\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") " pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:20.084041 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.083895    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwrb\" (UniqueName: \"kubernetes.io/projected/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-kube-api-access-xdwrb\") pod \"maas-controller-7f746cf4d4-kg8n2\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") " pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:20.184628 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.184593    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-metrics-certs\") pod \"maas-controller-7f746cf4d4-kg8n2\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") " pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:20.184827 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.184657    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-webhook-cert\") pod \"maas-controller-7f746cf4d4-kg8n2\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") " pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:20.184827 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.184678    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwrb\" (UniqueName: \"kubernetes.io/projected/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-kube-api-access-xdwrb\") pod \"maas-controller-7f746cf4d4-kg8n2\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") " pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:20.187173 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.187138    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-webhook-cert\") pod \"maas-controller-7f746cf4d4-kg8n2\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") " pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:20.187319 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.187195    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-metrics-certs\") pod \"maas-controller-7f746cf4d4-kg8n2\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") " pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:20.206291 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.206259    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwrb\" (UniqueName: \"kubernetes.io/projected/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-kube-api-access-xdwrb\") pod \"maas-controller-7f746cf4d4-kg8n2\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") " pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:20.296976 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.296910    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:20.605417 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.605326    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-kg8n2"]
Nov 02 13:50:20.609517 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:50:20.609483    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b3febd_e5d8_45ac_bc8c_944b268d2b2f.slice/crio-cd2f120705a66e60688becb4f3dfe0de70c1c982acb171f89b8b686203575956 WatchSource:0}: Error finding container cd2f120705a66e60688becb4f3dfe0de70c1c982acb171f89b8b686203575956: Status 404 returned error can't find the container with id cd2f120705a66e60688becb4f3dfe0de70c1c982acb171f89b8b686203575956
Nov 02 13:50:20.946616 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:20.946490    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2" event={"ID":"11b3febd-e5d8-45ac-bc8c-944b268d2b2f","Type":"ContainerStarted","Data":"cd2f120705a66e60688becb4f3dfe0de70c1c982acb171f89b8b686203575956"}
Nov 02 13:50:23.966283 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:23.966244    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2" event={"ID":"11b3febd-e5d8-45ac-bc8c-944b268d2b2f","Type":"ContainerStarted","Data":"abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b"}
Nov 02 13:50:23.966808 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:23.966346    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:24.008634 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:24.008573    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2" podStartSLOduration=2.66047744 podStartE2EDuration="5.008553801s" totalImagesPullingTime="2.348076361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:24.005074293 +0000 UTC m=+355.964449903" watchObservedRunningTime="2026-11-02 13:50:24.008553801 +0000 UTC m=+355.967929408"
Nov 02 13:50:24.928069 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:24.928032    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"]
Nov 02 13:50:24.977314 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:24.977274    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"]
Nov 02 13:50:24.977738 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:24.977398    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:24.996142 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:24.996107    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 02 13:50:25.004834 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.004799    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 02 13:50:25.005028 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.004848    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 02 13:50:25.005028 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.004848    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-x5c4m\""
Nov 02 13:50:25.033103 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.033060    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3a19987d-f187-4012-8627-c56ea413d6e7-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.033270 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.033225    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqxg\" (UniqueName: \"kubernetes.io/projected/3a19987d-f187-4012-8627-c56ea413d6e7-kube-api-access-9gqxg\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.033270 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.033259    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a19987d-f187-4012-8627-c56ea413d6e7-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.033349 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.033293    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/3a19987d-f187-4012-8627-c56ea413d6e7-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.134199 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.134161    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a19987d-f187-4012-8627-c56ea413d6e7-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.134412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.134207    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/3a19987d-f187-4012-8627-c56ea413d6e7-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.134412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.134262    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3a19987d-f187-4012-8627-c56ea413d6e7-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.134412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.134323    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqxg\" (UniqueName: \"kubernetes.io/projected/3a19987d-f187-4012-8627-c56ea413d6e7-kube-api-access-9gqxg\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.134986 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.134924    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a19987d-f187-4012-8627-c56ea413d6e7-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.135214 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.135190    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/3a19987d-f187-4012-8627-c56ea413d6e7-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.136736 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.136720    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3a19987d-f187-4012-8627-c56ea413d6e7-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.160177 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.160137    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqxg\" (UniqueName: \"kubernetes.io/projected/3a19987d-f187-4012-8627-c56ea413d6e7-kube-api-access-9gqxg\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-96kq9\" (UID: \"3a19987d-f187-4012-8627-c56ea413d6e7\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.325226 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.325187    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:25.706850 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.706763    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"]
Nov 02 13:50:25.710563 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:50:25.710529    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a19987d_f187_4012_8627_c56ea413d6e7.slice/crio-dffcc2a68f48574030e1a6766a19383ad9b6969b0c05b54e5321f73ddeb10817 WatchSource:0}: Error finding container dffcc2a68f48574030e1a6766a19383ad9b6969b0c05b54e5321f73ddeb10817: Status 404 returned error can't find the container with id dffcc2a68f48574030e1a6766a19383ad9b6969b0c05b54e5321f73ddeb10817
Nov 02 13:50:25.979536 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:25.979451    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9" event={"ID":"3a19987d-f187-4012-8627-c56ea413d6e7","Type":"ContainerStarted","Data":"dffcc2a68f48574030e1a6766a19383ad9b6969b0c05b54e5321f73ddeb10817"}
Nov 02 13:50:28.673529 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.671455    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-57f54"]
Nov 02 13:50:28.680081 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.680046    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-57f54"
Nov 02 13:50:28.694171 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.694139    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-57f54"]
Nov 02 13:50:28.698205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.698165    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 13:50:28.701441 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.701158    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xn892\""
Nov 02 13:50:28.720090 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.720048    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"]
Nov 02 13:50:28.724753 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.724723    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:50:28.750874 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.750838    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"]
Nov 02 13:50:28.775614 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.775575    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhnl\" (UniqueName: \"kubernetes.io/projected/88caf1c7-f64a-437c-a169-e6d629102ffc-kube-api-access-pjhnl\") pod \"kserve-controller-manager-776cc66d47-57f54\" (UID: \"88caf1c7-f64a-437c-a169-e6d629102ffc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-57f54"
Nov 02 13:50:28.775812 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.775721    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88caf1c7-f64a-437c-a169-e6d629102ffc-cert\") pod \"kserve-controller-manager-776cc66d47-57f54\" (UID: \"88caf1c7-f64a-437c-a169-e6d629102ffc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-57f54"
Nov 02 13:50:28.784563 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.784529    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cs5fh\""
Nov 02 13:50:28.784563 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.784532    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 02 13:50:28.823767 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.823727    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-zqrhs"]
Nov 02 13:50:28.828025 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.827995    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:28.845292 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.845257    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zqrhs"]
Nov 02 13:50:28.877187 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.876435    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhnl\" (UniqueName: \"kubernetes.io/projected/88caf1c7-f64a-437c-a169-e6d629102ffc-kube-api-access-pjhnl\") pod \"kserve-controller-manager-776cc66d47-57f54\" (UID: \"88caf1c7-f64a-437c-a169-e6d629102ffc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-57f54"
Nov 02 13:50:28.877187 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.876496    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrwfd\" (UniqueName: \"kubernetes.io/projected/17511bf5-8dbd-4a1c-990f-10032c0893e7-kube-api-access-jrwfd\") pod \"llmisvc-controller-manager-869d5dcc5d-mb4br\" (UID: \"17511bf5-8dbd-4a1c-990f-10032c0893e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:50:28.877187 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.876567    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17511bf5-8dbd-4a1c-990f-10032c0893e7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mb4br\" (UID: \"17511bf5-8dbd-4a1c-990f-10032c0893e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:50:28.877187 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.876650    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88caf1c7-f64a-437c-a169-e6d629102ffc-cert\") pod \"kserve-controller-manager-776cc66d47-57f54\" (UID: \"88caf1c7-f64a-437c-a169-e6d629102ffc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-57f54"
Nov 02 13:50:28.879766 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.879730    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88caf1c7-f64a-437c-a169-e6d629102ffc-cert\") pod \"kserve-controller-manager-776cc66d47-57f54\" (UID: \"88caf1c7-f64a-437c-a169-e6d629102ffc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-57f54"
Nov 02 13:50:28.898105 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.898070    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 02 13:50:28.898105 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.898086    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 02 13:50:28.898105 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.898109    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gph27\""
Nov 02 13:50:28.904379 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.904355    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhnl\" (UniqueName: \"kubernetes.io/projected/88caf1c7-f64a-437c-a169-e6d629102ffc-kube-api-access-pjhnl\") pod \"kserve-controller-manager-776cc66d47-57f54\" (UID: \"88caf1c7-f64a-437c-a169-e6d629102ffc\") " pod="opendatahub/kserve-controller-manager-776cc66d47-57f54"
Nov 02 13:50:28.978295 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.978212    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gskfk\" (UniqueName: \"kubernetes.io/projected/cc86d196-682a-4302-9892-6bbbf509bff0-kube-api-access-gskfk\") pod \"odh-model-controller-769875db56-zqrhs\" (UID: \"cc86d196-682a-4302-9892-6bbbf509bff0\") " pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:28.978444 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.978326    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrwfd\" (UniqueName: \"kubernetes.io/projected/17511bf5-8dbd-4a1c-990f-10032c0893e7-kube-api-access-jrwfd\") pod \"llmisvc-controller-manager-869d5dcc5d-mb4br\" (UID: \"17511bf5-8dbd-4a1c-990f-10032c0893e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:50:28.978444 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.978371    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc86d196-682a-4302-9892-6bbbf509bff0-metrics-certs\") pod \"odh-model-controller-769875db56-zqrhs\" (UID: \"cc86d196-682a-4302-9892-6bbbf509bff0\") " pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:28.978444 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.978424    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17511bf5-8dbd-4a1c-990f-10032c0893e7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mb4br\" (UID: \"17511bf5-8dbd-4a1c-990f-10032c0893e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:50:28.978566 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.978449    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc86d196-682a-4302-9892-6bbbf509bff0-cert\") pod \"odh-model-controller-769875db56-zqrhs\" (UID: \"cc86d196-682a-4302-9892-6bbbf509bff0\") " pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:28.982495 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:28.982441    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17511bf5-8dbd-4a1c-990f-10032c0893e7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-mb4br\" (UID: \"17511bf5-8dbd-4a1c-990f-10032c0893e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:50:29.000150 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.000116    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrwfd\" (UniqueName: \"kubernetes.io/projected/17511bf5-8dbd-4a1c-990f-10032c0893e7-kube-api-access-jrwfd\") pod \"llmisvc-controller-manager-869d5dcc5d-mb4br\" (UID: \"17511bf5-8dbd-4a1c-990f-10032c0893e7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:50:29.004382 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.004355    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-57f54"
Nov 02 13:50:29.046027 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.045990    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:50:29.079501 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.079457    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskfk\" (UniqueName: \"kubernetes.io/projected/cc86d196-682a-4302-9892-6bbbf509bff0-kube-api-access-gskfk\") pod \"odh-model-controller-769875db56-zqrhs\" (UID: \"cc86d196-682a-4302-9892-6bbbf509bff0\") " pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:29.079678 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.079558    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc86d196-682a-4302-9892-6bbbf509bff0-metrics-certs\") pod \"odh-model-controller-769875db56-zqrhs\" (UID: \"cc86d196-682a-4302-9892-6bbbf509bff0\") " pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:29.079678 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.079636    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc86d196-682a-4302-9892-6bbbf509bff0-cert\") pod \"odh-model-controller-769875db56-zqrhs\" (UID: \"cc86d196-682a-4302-9892-6bbbf509bff0\") " pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:29.085331 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.085259    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc86d196-682a-4302-9892-6bbbf509bff0-cert\") pod \"odh-model-controller-769875db56-zqrhs\" (UID: \"cc86d196-682a-4302-9892-6bbbf509bff0\") " pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:29.085579 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.085551    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc86d196-682a-4302-9892-6bbbf509bff0-metrics-certs\") pod \"odh-model-controller-769875db56-zqrhs\" (UID: \"cc86d196-682a-4302-9892-6bbbf509bff0\") " pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:29.117316 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.117277    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskfk\" (UniqueName: \"kubernetes.io/projected/cc86d196-682a-4302-9892-6bbbf509bff0-kube-api-access-gskfk\") pod \"odh-model-controller-769875db56-zqrhs\" (UID: \"cc86d196-682a-4302-9892-6bbbf509bff0\") " pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:29.144407 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.144366    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:29.322066 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.322025    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-57f54"]
Nov 02 13:50:29.371902 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.371861    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"]
Nov 02 13:50:29.468130 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:29.468090    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zqrhs"]
Nov 02 13:50:29.472752 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:50:29.472715    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc86d196_682a_4302_9892_6bbbf509bff0.slice/crio-606243c2ad17cb694891f76e63011aedff27d972ba1071307808d6ec669087d8 WatchSource:0}: Error finding container 606243c2ad17cb694891f76e63011aedff27d972ba1071307808d6ec669087d8: Status 404 returned error can't find the container with id 606243c2ad17cb694891f76e63011aedff27d972ba1071307808d6ec669087d8
Nov 02 13:50:30.010221 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:30.010172    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br" event={"ID":"17511bf5-8dbd-4a1c-990f-10032c0893e7","Type":"ContainerStarted","Data":"c5b1eefc91f1934685b882a673fa26b01f1a77eeb541ca63c57cfee75d0d5bb2"}
Nov 02 13:50:30.011855 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:30.011822    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zqrhs" event={"ID":"cc86d196-682a-4302-9892-6bbbf509bff0","Type":"ContainerStarted","Data":"606243c2ad17cb694891f76e63011aedff27d972ba1071307808d6ec669087d8"}
Nov 02 13:50:30.014090 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:30.014061    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-57f54" event={"ID":"88caf1c7-f64a-437c-a169-e6d629102ffc","Type":"ContainerStarted","Data":"a2365d0844be50d43924150ad3fe8af1aae766579bdda443a065f8fc1d0af20f"}
Nov 02 13:50:33.045902 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:33.045834    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-57f54" event={"ID":"88caf1c7-f64a-437c-a169-e6d629102ffc","Type":"ContainerStarted","Data":"fe04da417aab9c80cb2ebb62a365dcbda57d312713849c0cbf969bab62a6001e"}
Nov 02 13:50:33.046382 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:33.046153    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-57f54"
Nov 02 13:50:33.085688 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:33.085631    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-57f54" podStartSLOduration=1.499822442 podStartE2EDuration="5.085614286s" totalImagesPullingTime="3.585791844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:33.082369036 +0000 UTC m=+365.041744639" watchObservedRunningTime="2026-11-02 13:50:33.085614286 +0000 UTC m=+365.044989893"
Nov 02 13:50:34.054962 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:34.054884    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br" event={"ID":"17511bf5-8dbd-4a1c-990f-10032c0893e7","Type":"ContainerStarted","Data":"cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172"}
Nov 02 13:50:34.055513 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:34.054984    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:50:34.057228 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:34.057191    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zqrhs" event={"ID":"cc86d196-682a-4302-9892-6bbbf509bff0","Type":"ContainerStarted","Data":"e836000a85ad6ef3442dc2ebc26ab9e7698ac74c710c2898c920a8b6f0d39063"}
Nov 02 13:50:34.092111 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:34.090284    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br" podStartSLOduration=2.4715894130000002 podStartE2EDuration="6.090266367s" totalImagesPullingTime="3.618676954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:34.089754182 +0000 UTC m=+366.049129802" watchObservedRunningTime="2026-11-02 13:50:34.090266367 +0000 UTC m=+366.049641979"
Nov 02 13:50:34.126923 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:34.126863    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-zqrhs" podStartSLOduration=1.87939328 podStartE2EDuration="6.126843477s" totalImagesPullingTime="4.247450197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:34.126832536 +0000 UTC m=+366.086208143" watchObservedRunningTime="2026-11-02 13:50:34.126843477 +0000 UTC m=+366.086219086"
Nov 02 13:50:34.976390 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:34.976357    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 13:50:35.071190 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:35.071114    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:39.099029 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:39.098991    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9" event={"ID":"3a19987d-f187-4012-8627-c56ea413d6e7","Type":"ContainerStarted","Data":"64fa48b1690844df1243d7f04db9ac8ec8906630bdaeda327c6dd9e3043ebb50"}
Nov 02 13:50:39.099752 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:39.099187    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:50:39.135476 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:39.135414    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9" podStartSLOduration=2.618111665 podStartE2EDuration="15.135398023s" totalImagesPullingTime="12.517286358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:39.135006557 +0000 UTC m=+371.094382165" watchObservedRunningTime="2026-11-02 13:50:39.135398023 +0000 UTC m=+371.094773629"
Nov 02 13:50:43.894749 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:50:43.894710    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:50:46.078563 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:46.078531    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-zqrhs"
Nov 02 13:50:50.109749 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:50:50.109715    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-96kq9"
Nov 02 13:51:04.062675 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:04.062644    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-57f54"
Nov 02 13:51:04.872848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:04.872810    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"]
Nov 02 13:51:04.983535 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:04.983506    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"]
Nov 02 13:51:04.983708 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:04.983600    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.016768 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.016739    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17ae1c66-7df3-41ec-9e6f-94c76428ad59-openshift-service-ca\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.016891 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.016794    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-maas-api-tls\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.016891 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.016870    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-metrics-certs\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.017018 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.016964    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpt8s\" (UniqueName: \"kubernetes.io/projected/17ae1c66-7df3-41ec-9e6f-94c76428ad59-kube-api-access-hpt8s\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.060862 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.060840    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fnpxb\""
Nov 02 13:51:05.060862 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.060842    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 13:51:05.061047 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.060848    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 13:51:05.073841 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.073823    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:51:05.117691 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.117661    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpt8s\" (UniqueName: \"kubernetes.io/projected/17ae1c66-7df3-41ec-9e6f-94c76428ad59-kube-api-access-hpt8s\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.117809 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.117759    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17ae1c66-7df3-41ec-9e6f-94c76428ad59-openshift-service-ca\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.117874 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.117825    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-maas-api-tls\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.117874 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.117857    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-metrics-certs\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.118685 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.118663    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17ae1c66-7df3-41ec-9e6f-94c76428ad59-openshift-service-ca\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.120665 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.120645    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-maas-api-tls\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.121012 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.120903    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-metrics-certs\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.138927 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.138879    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpt8s\" (UniqueName: \"kubernetes.io/projected/17ae1c66-7df3-41ec-9e6f-94c76428ad59-kube-api-access-hpt8s\") pod \"maas-api-c77ff87df-7xkq6\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") " pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.320185 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.320156    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:05.642430 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:05.642399    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"]
Nov 02 13:51:05.646088 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:51:05.646057    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ae1c66_7df3_41ec_9e6f_94c76428ad59.slice/crio-8064595990caec1c377f4d3fb507c213d049805d7c17c34afaa1b1fcb6ec977f WatchSource:0}: Error finding container 8064595990caec1c377f4d3fb507c213d049805d7c17c34afaa1b1fcb6ec977f: Status 404 returned error can't find the container with id 8064595990caec1c377f4d3fb507c213d049805d7c17c34afaa1b1fcb6ec977f
Nov 02 13:51:06.235994 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:06.235953    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6" event={"ID":"17ae1c66-7df3-41ec-9e6f-94c76428ad59","Type":"ContainerStarted","Data":"8064595990caec1c377f4d3fb507c213d049805d7c17c34afaa1b1fcb6ec977f"}
Nov 02 13:51:06.740746 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:06.739917    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6lnnf"]
Nov 02 13:51:06.740746 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:06.740170    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf" podUID="9fe2507d-e0d9-4097-8d86-a5836a5be1be" containerName="authorino" containerID="cri-o://f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067" gracePeriod=30
Nov 02 13:51:07.034868 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.034846    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf"
Nov 02 13:51:07.138283 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.138256    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fe2507d-e0d9-4097-8d86-a5836a5be1be-kube-api-access-j29qb\" (UniqueName: \"kubernetes.io/projected/9fe2507d-e0d9-4097-8d86-a5836a5be1be-kube-api-access-j29qb\") pod \"9fe2507d-e0d9-4097-8d86-a5836a5be1be\" (UID: \"9fe2507d-e0d9-4097-8d86-a5836a5be1be\") "
Nov 02 13:51:07.140267 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.140234    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe2507d-e0d9-4097-8d86-a5836a5be1be-kube-api-access-j29qb" pod "9fe2507d-e0d9-4097-8d86-a5836a5be1be" (UID: "9fe2507d-e0d9-4097-8d86-a5836a5be1be"). InnerVolumeSpecName "kube-api-access-j29qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:51:07.239057 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.239024    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j29qb\" (UniqueName: \"kubernetes.io/projected/9fe2507d-e0d9-4097-8d86-a5836a5be1be-kube-api-access-j29qb\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:51:07.242917 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.242884    6284 generic.go:356] "Generic (PLEG): container finished" podID="9fe2507d-e0d9-4097-8d86-a5836a5be1be" containerID="f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067" exitCode=0
Nov 02 13:51:07.243080 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.242953    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf"
Nov 02 13:51:07.243080 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.242965    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf" event={"ID":"9fe2507d-e0d9-4097-8d86-a5836a5be1be","Type":"ContainerDied","Data":"f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067"}
Nov 02 13:51:07.243080 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.243024    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-6lnnf" event={"ID":"9fe2507d-e0d9-4097-8d86-a5836a5be1be","Type":"ContainerDied","Data":"41dc543e40c92af1f5ab30b36de6cc4c19517bccdbf76dcdf41879fe6028553a"}
Nov 02 13:51:07.243080 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.243048    6284 scope.go:122] "RemoveContainer" containerID="f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067"
Nov 02 13:51:07.254572 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.254557    6284 scope.go:122] "RemoveContainer" containerID="f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067"
Nov 02 13:51:07.255277 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:51:07.255256    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067\": container with ID starting with f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067 not found: ID does not exist" containerID="f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067"
Nov 02 13:51:07.255384 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.255284    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067"} err="failed to get container status \"f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067\": rpc error: code = NotFound desc = could not find container \"f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067\": container with ID starting with f87dfb6f0646eb4c931f2ba75972e3a4a35567b04b65b90b39bdccfd4c45d067 not found: ID does not exist"
Nov 02 13:51:07.286952 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.286914    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6lnnf"]
Nov 02 13:51:07.291374 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:07.291247    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-6lnnf"]
Nov 02 13:51:08.173819 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:08.173784    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fe2507d-e0d9-4097-8d86-a5836a5be1be" path="/var/lib/kubelet/pods/9fe2507d-e0d9-4097-8d86-a5836a5be1be/volumes"
Nov 02 13:51:08.249678 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:08.249638    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6" event={"ID":"17ae1c66-7df3-41ec-9e6f-94c76428ad59","Type":"ContainerStarted","Data":"cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950"}
Nov 02 13:51:08.250154 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:08.249810    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:08.289323 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:08.289278    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6" podStartSLOduration=2.363650204 podStartE2EDuration="4.28926292s" totalImagesPullingTime="1.925612716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:51:08.287303141 +0000 UTC m=+400.246678748" watchObservedRunningTime="2026-11-02 13:51:08.28926292 +0000 UTC m=+400.248638526"
Nov 02 13:51:13.894055 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:51:13.894012    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:51:14.261769 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:14.261740    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 13:51:27.784555 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:27.784518    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"]
Nov 02 13:51:27.784983 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:27.784949    6284 cpu_manager.go:414] "containerMap: removing container" podUID="9fe2507d-e0d9-4097-8d86-a5836a5be1be" containerName="authorino"
Nov 02 13:51:27.784983 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:27.784962    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fe2507d-e0d9-4097-8d86-a5836a5be1be" containerName="authorino"
Nov 02 13:51:27.785060 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:27.785039    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fe2507d-e0d9-4097-8d86-a5836a5be1be" containerName="authorino"
Nov 02 13:51:27.788287 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:27.788270    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"
Nov 02 13:51:27.808498 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:27.808473    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"]
Nov 02 13:51:27.922598 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:27.922571    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwz7\" (UniqueName: \"kubernetes.io/projected/3f60483b-1b89-404c-8782-8efca76bb8d1-kube-api-access-xqwz7\") pod \"llmisvc-controller-manager-6466c554d8-j74b9\" (UID: \"3f60483b-1b89-404c-8782-8efca76bb8d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"
Nov 02 13:51:27.922752 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:27.922606    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f60483b-1b89-404c-8782-8efca76bb8d1-cert\") pod \"llmisvc-controller-manager-6466c554d8-j74b9\" (UID: \"3f60483b-1b89-404c-8782-8efca76bb8d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"
Nov 02 13:51:28.023878 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:28.023847    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwz7\" (UniqueName: \"kubernetes.io/projected/3f60483b-1b89-404c-8782-8efca76bb8d1-kube-api-access-xqwz7\") pod \"llmisvc-controller-manager-6466c554d8-j74b9\" (UID: \"3f60483b-1b89-404c-8782-8efca76bb8d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"
Nov 02 13:51:28.024030 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:28.023887    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f60483b-1b89-404c-8782-8efca76bb8d1-cert\") pod \"llmisvc-controller-manager-6466c554d8-j74b9\" (UID: \"3f60483b-1b89-404c-8782-8efca76bb8d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"
Nov 02 13:51:28.026436 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:28.026414    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f60483b-1b89-404c-8782-8efca76bb8d1-cert\") pod \"llmisvc-controller-manager-6466c554d8-j74b9\" (UID: \"3f60483b-1b89-404c-8782-8efca76bb8d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"
Nov 02 13:51:28.050374 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:28.050313    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwz7\" (UniqueName: \"kubernetes.io/projected/3f60483b-1b89-404c-8782-8efca76bb8d1-kube-api-access-xqwz7\") pod \"llmisvc-controller-manager-6466c554d8-j74b9\" (UID: \"3f60483b-1b89-404c-8782-8efca76bb8d1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"
Nov 02 13:51:28.101029 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:28.101008    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"
Nov 02 13:51:28.394109 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:28.394040    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"]
Nov 02 13:51:28.397741 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:51:28.397716    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f60483b_1b89_404c_8782_8efca76bb8d1.slice/crio-beafd53e95de64067c26b9f328784068532b222f261a6ecbc7d87b26bc0ae575 WatchSource:0}: Error finding container beafd53e95de64067c26b9f328784068532b222f261a6ecbc7d87b26bc0ae575: Status 404 returned error can't find the container with id beafd53e95de64067c26b9f328784068532b222f261a6ecbc7d87b26bc0ae575
Nov 02 13:51:29.352917 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:29.352880    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9" event={"ID":"3f60483b-1b89-404c-8782-8efca76bb8d1","Type":"ContainerStarted","Data":"75217841a368f248dd28365ce29dc4f55d393c6d87fdd3bbf482fed9d11b4bf0"}
Nov 02 13:51:29.353330 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:29.352958    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9" event={"ID":"3f60483b-1b89-404c-8782-8efca76bb8d1","Type":"ContainerStarted","Data":"beafd53e95de64067c26b9f328784068532b222f261a6ecbc7d87b26bc0ae575"}
Nov 02 13:51:29.353330 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:29.352989    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"
Nov 02 13:51:29.390609 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:51:29.390553    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9" podStartSLOduration=2.10465968 podStartE2EDuration="2.39053327s" totalImagesPullingTime="285.87359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:51:29.388836268 +0000 UTC m=+421.348211874" watchObservedRunningTime="2026-11-02 13:51:29.39053327 +0000 UTC m=+421.349908879"
Nov 02 13:51:43.892809 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:51:43.892774    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:52:00.358687 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:00.358611    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-j74b9"
Nov 02 13:52:00.427903 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:00.427870    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"]
Nov 02 13:52:00.428175 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:00.428126    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br" podUID="17511bf5-8dbd-4a1c-990f-10032c0893e7" containerName="manager" containerID="cri-o://cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172" gracePeriod=30
Nov 02 13:52:00.663433 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:00.663407    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:52:00.675765 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:00.675739    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17511bf5-8dbd-4a1c-990f-10032c0893e7-cert\" (UniqueName: \"kubernetes.io/secret/17511bf5-8dbd-4a1c-990f-10032c0893e7-cert\") pod \"17511bf5-8dbd-4a1c-990f-10032c0893e7\" (UID: \"17511bf5-8dbd-4a1c-990f-10032c0893e7\") "
Nov 02 13:52:00.675884 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:00.675807    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17511bf5-8dbd-4a1c-990f-10032c0893e7-kube-api-access-jrwfd\" (UniqueName: \"kubernetes.io/projected/17511bf5-8dbd-4a1c-990f-10032c0893e7-kube-api-access-jrwfd\") pod \"17511bf5-8dbd-4a1c-990f-10032c0893e7\" (UID: \"17511bf5-8dbd-4a1c-990f-10032c0893e7\") "
Nov 02 13:52:00.677829 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:00.677806    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17511bf5-8dbd-4a1c-990f-10032c0893e7-kube-api-access-jrwfd" pod "17511bf5-8dbd-4a1c-990f-10032c0893e7" (UID: "17511bf5-8dbd-4a1c-990f-10032c0893e7"). InnerVolumeSpecName "kube-api-access-jrwfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:52:00.677914 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:00.677839    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17511bf5-8dbd-4a1c-990f-10032c0893e7-cert" pod "17511bf5-8dbd-4a1c-990f-10032c0893e7" (UID: "17511bf5-8dbd-4a1c-990f-10032c0893e7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:52:00.776922 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:00.776894    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrwfd\" (UniqueName: \"kubernetes.io/projected/17511bf5-8dbd-4a1c-990f-10032c0893e7-kube-api-access-jrwfd\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:52:00.776922 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:00.776919    6284 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17511bf5-8dbd-4a1c-990f-10032c0893e7-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:52:01.515091 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:01.515052    6284 generic.go:356] "Generic (PLEG): container finished" podID="17511bf5-8dbd-4a1c-990f-10032c0893e7" containerID="cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172" exitCode=0
Nov 02 13:52:01.515560 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:01.515100    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"
Nov 02 13:52:01.515560 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:01.515127    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br" event={"ID":"17511bf5-8dbd-4a1c-990f-10032c0893e7","Type":"ContainerDied","Data":"cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172"}
Nov 02 13:52:01.515560 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:01.515180    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br" event={"ID":"17511bf5-8dbd-4a1c-990f-10032c0893e7","Type":"ContainerDied","Data":"c5b1eefc91f1934685b882a673fa26b01f1a77eeb541ca63c57cfee75d0d5bb2"}
Nov 02 13:52:01.515560 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:01.515203    6284 scope.go:122] "RemoveContainer" containerID="cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172"
Nov 02 13:52:01.525252 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:01.525229    6284 scope.go:122] "RemoveContainer" containerID="cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172"
Nov 02 13:52:01.525555 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:52:01.525537    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172\": container with ID starting with cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172 not found: ID does not exist" containerID="cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172"
Nov 02 13:52:01.525611 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:01.525563    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172"} err="failed to get container status \"cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172\": rpc error: code = NotFound desc = could not find container \"cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172\": container with ID starting with cd34bf1fd6b48f5f02ec5aa4d3def86ffa0a683ae965eb5c82a69f95255cb172 not found: ID does not exist"
Nov 02 13:52:01.570367 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:01.570332    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"]
Nov 02 13:52:01.580104 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:01.580084    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-mb4br"]
Nov 02 13:52:02.168556 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:52:02.168528    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17511bf5-8dbd-4a1c-990f-10032c0893e7" path="/var/lib/kubelet/pods/17511bf5-8dbd-4a1c-990f-10032c0893e7/volumes"
Nov 02 13:52:13.892333 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:52:13.892295    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:52:43.892735 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:52:43.892700    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:53:00.257870 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.257842    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"]
Nov 02 13:53:00.260290 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.258454    6284 cpu_manager.go:414] "containerMap: removing container" podUID="17511bf5-8dbd-4a1c-990f-10032c0893e7" containerName="manager"
Nov 02 13:53:00.260290 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.258469    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17511bf5-8dbd-4a1c-990f-10032c0893e7" containerName="manager"
Nov 02 13:53:00.260290 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.258532    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="17511bf5-8dbd-4a1c-990f-10032c0893e7" containerName="manager"
Nov 02 13:53:00.261421 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.261400    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.283464 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.283437    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 13:53:00.291562 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.291528    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"]
Nov 02 13:53:00.291923 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.291890    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.292163 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.291928    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.292163 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.291969    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.292163 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.292072    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.292303 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.292172    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n7s2\" (UniqueName: \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-kube-api-access-9n7s2\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.292303 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.292205    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.292303 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.292223    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.292303 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.292263    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.292303 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.292283    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.320790 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.320764    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-27gvt\""
Nov 02 13:53:00.392899 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.392875    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393023 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.392906    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393023 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.392973    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393023 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.392992    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393023 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.393014    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393202 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.393043    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393202 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.393070    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n7s2\" (UniqueName: \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-kube-api-access-9n7s2\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393202 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.393102    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393202 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.393130    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393547 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.393521    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393644 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.393582    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.393652    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.393694    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.393927 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.393904    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.395286 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.395266    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.395513 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.395497    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.413759 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.413733    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.413841 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.413827    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n7s2\" (UniqueName: \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-kube-api-access-9n7s2\") pod \"maas-default-gateway-openshift-default-6886b96f9-hwr8w\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.577171 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.577096    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:00.890186 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.890118    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"]
Nov 02 13:53:00.893631 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:53:00.893604    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323cdaf7_0099_49fe_ac81_d64d7acd09c0.slice/crio-5e4361fdfc60f4e0065edade1b0af5344f68b4318926e40e2c393e79f576c951 WatchSource:0}: Error finding container 5e4361fdfc60f4e0065edade1b0af5344f68b4318926e40e2c393e79f576c951: Status 404 returned error can't find the container with id 5e4361fdfc60f4e0065edade1b0af5344f68b4318926e40e2c393e79f576c951
Nov 02 13:53:00.895796 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:00.895770    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:53:01.833578 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:01.833545    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" event={"ID":"323cdaf7-0099-49fe-ac81-d64d7acd09c0","Type":"ContainerStarted","Data":"5e4361fdfc60f4e0065edade1b0af5344f68b4318926e40e2c393e79f576c951"}
Nov 02 13:53:04.397368 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:04.397342    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:53:04.397610 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:04.397416    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:53:04.397610 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:04.397454    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:53:04.878976 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:04.878901    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" event={"ID":"323cdaf7-0099-49fe-ac81-d64d7acd09c0","Type":"ContainerStarted","Data":"f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc"}
Nov 02 13:53:04.940185 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:04.940118    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podStartSLOduration=1.439667563 podStartE2EDuration="4.940103715s" totalImagesPullingTime="3.500436152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:04.929311094 +0000 UTC m=+516.888686702" watchObservedRunningTime="2026-11-02 13:53:04.940103715 +0000 UTC m=+516.899479319"
Nov 02 13:53:05.578356 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:05.578316    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:05.579987 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:05.579960    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hwr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 13:53:05.580111 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:05.580018    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 13:53:06.580378 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:06.580342    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hwr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 13:53:06.580891 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:06.580416    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 13:53:07.577601 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:07.577566    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hwr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 13:53:07.577787 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:07.577637    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 13:53:08.580375 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:08.580333    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hwr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 13:53:08.580801 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:08.580400    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 13:53:09.578362 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:09.578322    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hwr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 13:53:09.578569 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:09.578385    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 13:53:10.577910 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:10.577867    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hwr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 13:53:10.578327 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:10.577961    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 13:53:11.578381 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:11.578342    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hwr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Nov 02 13:53:11.578848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:11.578414    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Nov 02 13:53:12.582127 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:12.582099    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:12.582585 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:12.582551    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:12.583243 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:12.583226    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 13:53:13.892099 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:53:13.892064    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:53:15.298541 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.298511    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"]
Nov 02 13:53:15.301965 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.301949    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.328279 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.328248    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"]
Nov 02 13:53:15.445619 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.445587    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.445619 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.445620    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rq4b\" (UniqueName: \"kubernetes.io/projected/eb79c248-2d1c-42a1-82dd-57d1545d5588-kube-api-access-6rq4b\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.445819 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.445655    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.445819 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.445784    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.445896 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.445845    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.446001 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.445980    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.446039 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.446009    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.446073 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.446044    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb79c248-2d1c-42a1-82dd-57d1545d5588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.446114 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.446102    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.547509 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.547476    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb79c248-2d1c-42a1-82dd-57d1545d5588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.547509 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.547515    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.547749 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.547545    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.547749 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.547574    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rq4b\" (UniqueName: \"kubernetes.io/projected/eb79c248-2d1c-42a1-82dd-57d1545d5588-kube-api-access-6rq4b\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.547749 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.547614    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.547749 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.547644    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.547749 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.547706    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.548032 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.548002    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.548093 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.548039    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.548093 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.548079    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.548203 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.548116    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.548203 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.548146    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.548309 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.548290    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.548428 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.548408    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb79c248-2d1c-42a1-82dd-57d1545d5588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.550134 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.550081    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.550268 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.550253    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.568619 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.568590    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb79c248-2d1c-42a1-82dd-57d1545d5588-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.568821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.568801    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rq4b\" (UniqueName: \"kubernetes.io/projected/eb79c248-2d1c-42a1-82dd-57d1545d5588-kube-api-access-6rq4b\") pod \"maas-default-gateway-openshift-default-6886b96f9-49rxk\" (UID: \"eb79c248-2d1c-42a1-82dd-57d1545d5588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.615456 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.615431    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:15.938399 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.938324    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"]
Nov 02 13:53:15.942629 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:53:15.942604    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb79c248_2d1c_42a1_82dd_57d1545d5588.slice/crio-9014316696c4e0a5b4afea98066e119416fd6fab165bd3b2f28c512a52c3046a WatchSource:0}: Error finding container 9014316696c4e0a5b4afea98066e119416fd6fab165bd3b2f28c512a52c3046a: Status 404 returned error can't find the container with id 9014316696c4e0a5b4afea98066e119416fd6fab165bd3b2f28c512a52c3046a
Nov 02 13:53:15.945541 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.945511    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:53:15.945621 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.945575    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:53:15.945621 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:15.945602    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:53:16.939142 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:16.939109    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" event={"ID":"eb79c248-2d1c-42a1-82dd-57d1545d5588","Type":"ContainerStarted","Data":"c3bc638de659857cbbb17b36074d2948a58b6a325d66254dd45f89321f5d1c2c"}
Nov 02 13:53:16.939528 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:16.939160    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" event={"ID":"eb79c248-2d1c-42a1-82dd-57d1545d5588","Type":"ContainerStarted","Data":"9014316696c4e0a5b4afea98066e119416fd6fab165bd3b2f28c512a52c3046a"}
Nov 02 13:53:17.007957 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:17.007885    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podStartSLOduration=2.007871706 podStartE2EDuration="2.007871706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:16.996967694 +0000 UTC m=+528.956343297" watchObservedRunningTime="2026-11-02 13:53:17.007871706 +0000 UTC m=+528.967247314"
Nov 02 13:53:17.616129 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:17.616093    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:17.617578 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:17.617552    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 13:53:17.617706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:17.617600    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 13:53:18.616270 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:18.616234    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 13:53:18.616699 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:18.616287    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 13:53:19.615908 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:19.615873    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 13:53:19.616108 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:19.615960    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 13:53:20.616624 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:20.616590    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 13:53:20.617039 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:20.616660    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 13:53:21.616595 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:21.616560    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 13:53:21.616782 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:21.616624    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 13:53:22.616533 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:22.616497    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Nov 02 13:53:22.616714 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:22.616548    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Nov 02 13:53:24.139703 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:24.139670    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:24.140245 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:24.140219    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:24.140991 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:24.140970    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk"
Nov 02 13:53:43.896753 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:53:43.896710    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:53:47.671837 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.671798    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"]
Nov 02 13:53:47.681276 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.681246    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.707206 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.707165    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"]
Nov 02 13:53:47.707393 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.707351    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ebfc3db5-serving-cert\""
Nov 02 13:53:47.781747 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.781705    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-metrics-certs\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.781992 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.781876    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-maas-api-tls\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.781992 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.781912    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ssx\" (UniqueName: \"kubernetes.io/projected/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-kube-api-access-p8ssx\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.782133 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.782038    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-openshift-service-ca\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.883159 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.883113    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-metrics-certs\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.883351 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.883207    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-maas-api-tls\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.883351 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.883225    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ssx\" (UniqueName: \"kubernetes.io/projected/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-kube-api-access-p8ssx\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.883351 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.883257    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-openshift-service-ca\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.883465 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:53:47.883370    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-serving-cert: secret "maas-api-e2e-ait-ebfc3db5-serving-cert" not found
Nov 02 13:53:47.883465 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:53:47.883463    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-maas-api-tls podName:ccb369c9-bc1b-48c7-9a18-0b021d9952b3 nodeName:}" failed. No retries permitted until 2026-11-02 13:53:48.383444938 +0000 UTC m=+560.342820545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-maas-api-tls") pod "maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24" (UID: "ccb369c9-bc1b-48c7-9a18-0b021d9952b3") : secret "maas-api-e2e-ait-ebfc3db5-serving-cert" not found
Nov 02 13:53:47.884059 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.884040    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-openshift-service-ca\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.885661 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.885634    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-metrics-certs\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:47.906827 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:47.906789    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ssx\" (UniqueName: \"kubernetes.io/projected/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-kube-api-access-p8ssx\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:48.387724 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:48.387686    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-maas-api-tls\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:48.390623 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:48.390592    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-maas-api-tls\") pod \"maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:48.596909 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:48.596866    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:48.917322 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:48.917212    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"]
Nov 02 13:53:49.146457 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:49.146423    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24" event={"ID":"ccb369c9-bc1b-48c7-9a18-0b021d9952b3","Type":"ContainerStarted","Data":"2fc0817e70f6eb9d000c13bc954280c0395b867a765c6dde4092337f0dced974"}
Nov 02 13:53:49.856329 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:49.856274    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"]
Nov 02 13:53:49.860965 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:49.860918    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:49.875873 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:49.875844    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3c148ffa-serving-cert\""
Nov 02 13:53:49.891578 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:49.891541    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"]
Nov 02 13:53:50.006394 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.006358    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c6d773-504b-4b56-ad89-5953df286c3e-openshift-service-ca\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.006831 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.006442    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-maas-api-tls\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.006831 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.006526    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjg2x\" (UniqueName: \"kubernetes.io/projected/10c6d773-504b-4b56-ad89-5953df286c3e-kube-api-access-bjg2x\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.006831 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.006561    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-metrics-certs\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.108194 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.108088    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c6d773-504b-4b56-ad89-5953df286c3e-openshift-service-ca\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.108194 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.108184    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-maas-api-tls\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.108440 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.108224    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjg2x\" (UniqueName: \"kubernetes.io/projected/10c6d773-504b-4b56-ad89-5953df286c3e-kube-api-access-bjg2x\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.108440 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.108262    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-metrics-certs\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.108440 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:53:50.108356    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-serving-cert: secret "maas-api-e2e-ait-3c148ffa-serving-cert" not found
Nov 02 13:53:50.108440 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:53:50.108434    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-maas-api-tls podName:10c6d773-504b-4b56-ad89-5953df286c3e nodeName:}" failed. No retries permitted until 2026-11-02 13:53:50.60841425 +0000 UTC m=+562.567789849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-maas-api-tls") pod "maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv" (UID: "10c6d773-504b-4b56-ad89-5953df286c3e") : secret "maas-api-e2e-ait-3c148ffa-serving-cert" not found
Nov 02 13:53:50.108915 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.108891    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c6d773-504b-4b56-ad89-5953df286c3e-openshift-service-ca\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.111002 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.110976    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-metrics-certs\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.135404 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.135369    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjg2x\" (UniqueName: \"kubernetes.io/projected/10c6d773-504b-4b56-ad89-5953df286c3e-kube-api-access-bjg2x\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.153758 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.153718    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24" event={"ID":"ccb369c9-bc1b-48c7-9a18-0b021d9952b3","Type":"ContainerStarted","Data":"e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238"}
Nov 02 13:53:50.154059 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.153865    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:50.195604 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.195535    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24" podStartSLOduration=2.828861645 podStartE2EDuration="3.195510759s" totalImagesPullingTime="366.649114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:50.193952036 +0000 UTC m=+562.153327643" watchObservedRunningTime="2026-11-02 13:53:50.195510759 +0000 UTC m=+562.154886366"
Nov 02 13:53:50.614459 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.614415    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-maas-api-tls\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.616925 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.616892    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-maas-api-tls\") pod \"maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:50.776921 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:50.776878    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:51.085644 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:51.085605    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"]
Nov 02 13:53:51.087576 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:53:51.087535    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c6d773_504b_4b56_ad89_5953df286c3e.slice/crio-c3b82d900ea7462521ee1dbb01a4dc826dea172ebd57209d690ad5b5bbdd7167 WatchSource:0}: Error finding container c3b82d900ea7462521ee1dbb01a4dc826dea172ebd57209d690ad5b5bbdd7167: Status 404 returned error can't find the container with id c3b82d900ea7462521ee1dbb01a4dc826dea172ebd57209d690ad5b5bbdd7167
Nov 02 13:53:51.162833 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:51.162784    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv" event={"ID":"10c6d773-504b-4b56-ad89-5953df286c3e","Type":"ContainerStarted","Data":"c3b82d900ea7462521ee1dbb01a4dc826dea172ebd57209d690ad5b5bbdd7167"}
Nov 02 13:53:52.171036 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:52.171001    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv" event={"ID":"10c6d773-504b-4b56-ad89-5953df286c3e","Type":"ContainerStarted","Data":"8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a"}
Nov 02 13:53:52.171574 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:52.171097    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:53:52.215637 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:52.215578    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv" podStartSLOduration=2.897709669 podStartE2EDuration="3.215561313s" totalImagesPullingTime="317.851644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:52.214888738 +0000 UTC m=+564.174264347" watchObservedRunningTime="2026-11-02 13:53:52.215561313 +0000 UTC m=+564.174936919"
Nov 02 13:53:54.002307 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.002272    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"]
Nov 02 13:53:54.006092 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.006061    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.045818 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.045775    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"]
Nov 02 13:53:54.095759 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.095726    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c57ba6aa-serving-cert\""
Nov 02 13:53:54.153565 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.153519    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ba0dbe-ef60-4c3e-9153-b06987ab6213-openshift-service-ca\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.153757 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.153594    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-metrics-certs\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.153757 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.153732    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2drg\" (UniqueName: \"kubernetes.io/projected/59ba0dbe-ef60-4c3e-9153-b06987ab6213-kube-api-access-b2drg\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.153851 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.153800    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-maas-api-tls\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.254764 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.254664    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ba0dbe-ef60-4c3e-9153-b06987ab6213-openshift-service-ca\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.254764 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.254712    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-metrics-certs\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.254764 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.254754    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2drg\" (UniqueName: \"kubernetes.io/projected/59ba0dbe-ef60-4c3e-9153-b06987ab6213-kube-api-access-b2drg\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.255014 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.254781    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-maas-api-tls\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.255520 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.255496    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ba0dbe-ef60-4c3e-9153-b06987ab6213-openshift-service-ca\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.257424 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.257394    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-maas-api-tls\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.257424 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.257412    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-metrics-certs\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.294199 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.294167    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2drg\" (UniqueName: \"kubernetes.io/projected/59ba0dbe-ef60-4c3e-9153-b06987ab6213-kube-api-access-b2drg\") pod \"maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.322016 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.321978    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:54.652845 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:54.652742    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"]
Nov 02 13:53:54.656478 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:53:54.656449    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ba0dbe_ef60_4c3e_9153_b06987ab6213.slice/crio-822d76ddb4116791a6994c34ccd94d52c7a786ed444f999e3bef888c87b3565c WatchSource:0}: Error finding container 822d76ddb4116791a6994c34ccd94d52c7a786ed444f999e3bef888c87b3565c: Status 404 returned error can't find the container with id 822d76ddb4116791a6994c34ccd94d52c7a786ed444f999e3bef888c87b3565c
Nov 02 13:53:55.190751 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:55.190712    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr" event={"ID":"59ba0dbe-ef60-4c3e-9153-b06987ab6213","Type":"ContainerStarted","Data":"fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86"}
Nov 02 13:53:55.191295 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:55.190773    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr" event={"ID":"59ba0dbe-ef60-4c3e-9153-b06987ab6213","Type":"ContainerStarted","Data":"822d76ddb4116791a6994c34ccd94d52c7a786ed444f999e3bef888c87b3565c"}
Nov 02 13:53:55.191295 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:55.190799    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:53:56.177293 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:56.177267    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:53:56.230354 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:56.230286    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr" podStartSLOduration=2.92798385 podStartE2EDuration="3.230268885s" totalImagesPullingTime="302.285035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:55.246397536 +0000 UTC m=+567.205773144" watchObservedRunningTime="2026-11-02 13:53:56.230268885 +0000 UTC m=+568.189644492"
Nov 02 13:53:58.184071 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:53:58.184036    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:54:01.202531 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:01.202498    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:54:13.895319 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:54:13.895281    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:54:21.311018 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.310973    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"]
Nov 02 13:54:21.311755 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.311249    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24" podUID="ccb369c9-bc1b-48c7-9a18-0b021d9952b3" containerName="maas-api" containerID="cri-o://e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238" gracePeriod=30
Nov 02 13:54:21.576305 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.576222    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:54:21.742647 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.742609    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-metrics-certs\") pod \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") "
Nov 02 13:54:21.742879 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.742687    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-maas-api-tls\") pod \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") "
Nov 02 13:54:21.742879 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.742722    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-kube-api-access-p8ssx\" (UniqueName: \"kubernetes.io/projected/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-kube-api-access-p8ssx\") pod \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") "
Nov 02 13:54:21.742879 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.742753    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-openshift-service-ca\") pod \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\" (UID: \"ccb369c9-bc1b-48c7-9a18-0b021d9952b3\") "
Nov 02 13:54:21.743508 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.743466    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-openshift-service-ca" pod "ccb369c9-bc1b-48c7-9a18-0b021d9952b3" (UID: "ccb369c9-bc1b-48c7-9a18-0b021d9952b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:21.746253 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.746208    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-maas-api-tls" pod "ccb369c9-bc1b-48c7-9a18-0b021d9952b3" (UID: "ccb369c9-bc1b-48c7-9a18-0b021d9952b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:21.746415 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.746282    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-metrics-certs" pod "ccb369c9-bc1b-48c7-9a18-0b021d9952b3" (UID: "ccb369c9-bc1b-48c7-9a18-0b021d9952b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:21.746415 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.746405    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-kube-api-access-p8ssx" pod "ccb369c9-bc1b-48c7-9a18-0b021d9952b3" (UID: "ccb369c9-bc1b-48c7-9a18-0b021d9952b3"). InnerVolumeSpecName "kube-api-access-p8ssx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:21.844592 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.844486    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:21.844592 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.844525    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:21.844592 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.844538    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ssx\" (UniqueName: \"kubernetes.io/projected/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-kube-api-access-p8ssx\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:21.844592 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:21.844551    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccb369c9-bc1b-48c7-9a18-0b021d9952b3-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:22.347387 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:22.347351    6284 generic.go:356] "Generic (PLEG): container finished" podID="ccb369c9-bc1b-48c7-9a18-0b021d9952b3" containerID="e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238" exitCode=0
Nov 02 13:54:22.347980 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:22.347398    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"
Nov 02 13:54:22.347980 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:22.347442    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24" event={"ID":"ccb369c9-bc1b-48c7-9a18-0b021d9952b3","Type":"ContainerDied","Data":"e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238"}
Nov 02 13:54:22.347980 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:22.347510    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24" event={"ID":"ccb369c9-bc1b-48c7-9a18-0b021d9952b3","Type":"ContainerDied","Data":"2fc0817e70f6eb9d000c13bc954280c0395b867a765c6dde4092337f0dced974"}
Nov 02 13:54:22.347980 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:22.347542    6284 scope.go:122] "RemoveContainer" containerID="e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238"
Nov 02 13:54:22.357429 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:22.357380    6284 scope.go:122] "RemoveContainer" containerID="e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238"
Nov 02 13:54:22.357860 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:54:22.357838    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238\": container with ID starting with e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238 not found: ID does not exist" containerID="e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238"
Nov 02 13:54:22.358017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:22.357877    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238"} err="failed to get container status \"e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238\": rpc error: code = NotFound desc = could not find container \"e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238\": container with ID starting with e0fe58e2273e14d9ed84fa149869c95ead3ca1eb52230c7cca8b5cd6338f5238 not found: ID does not exist"
Nov 02 13:54:22.393718 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:22.393684    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"]
Nov 02 13:54:22.403067 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:22.403030    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebfc3db5-676fdd759-h8b24"]
Nov 02 13:54:23.850742 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:23.850706    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"]
Nov 02 13:54:23.851222 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:23.850921    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv" podUID="10c6d773-504b-4b56-ad89-5953df286c3e" containerName="maas-api" containerID="cri-o://8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a" gracePeriod=30
Nov 02 13:54:24.144016 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.143987    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:54:24.168457 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.168411    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb369c9-bc1b-48c7-9a18-0b021d9952b3" path="/var/lib/kubelet/pods/ccb369c9-bc1b-48c7-9a18-0b021d9952b3/volumes"
Nov 02 13:54:24.270013 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.269963    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c6d773-504b-4b56-ad89-5953df286c3e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c6d773-504b-4b56-ad89-5953df286c3e-openshift-service-ca\") pod \"10c6d773-504b-4b56-ad89-5953df286c3e\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") "
Nov 02 13:54:24.270013 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.270010    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-metrics-certs\") pod \"10c6d773-504b-4b56-ad89-5953df286c3e\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") "
Nov 02 13:54:24.270348 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.270085    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c6d773-504b-4b56-ad89-5953df286c3e-kube-api-access-bjg2x\" (UniqueName: \"kubernetes.io/projected/10c6d773-504b-4b56-ad89-5953df286c3e-kube-api-access-bjg2x\") pod \"10c6d773-504b-4b56-ad89-5953df286c3e\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") "
Nov 02 13:54:24.270348 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.270124    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-maas-api-tls\") pod \"10c6d773-504b-4b56-ad89-5953df286c3e\" (UID: \"10c6d773-504b-4b56-ad89-5953df286c3e\") "
Nov 02 13:54:24.270472 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.270433    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c6d773-504b-4b56-ad89-5953df286c3e-openshift-service-ca" pod "10c6d773-504b-4b56-ad89-5953df286c3e" (UID: "10c6d773-504b-4b56-ad89-5953df286c3e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:24.272528 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.272490    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-maas-api-tls" pod "10c6d773-504b-4b56-ad89-5953df286c3e" (UID: "10c6d773-504b-4b56-ad89-5953df286c3e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:24.272783 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.272768    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-metrics-certs" pod "10c6d773-504b-4b56-ad89-5953df286c3e" (UID: "10c6d773-504b-4b56-ad89-5953df286c3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:24.272891 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.272867    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c6d773-504b-4b56-ad89-5953df286c3e-kube-api-access-bjg2x" pod "10c6d773-504b-4b56-ad89-5953df286c3e" (UID: "10c6d773-504b-4b56-ad89-5953df286c3e"). InnerVolumeSpecName "kube-api-access-bjg2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:24.366911 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.366871    6284 generic.go:356] "Generic (PLEG): container finished" podID="10c6d773-504b-4b56-ad89-5953df286c3e" containerID="8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a" exitCode=0
Nov 02 13:54:24.367316 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.367030    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv" event={"ID":"10c6d773-504b-4b56-ad89-5953df286c3e","Type":"ContainerDied","Data":"8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a"}
Nov 02 13:54:24.367517 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.367497    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv" event={"ID":"10c6d773-504b-4b56-ad89-5953df286c3e","Type":"ContainerDied","Data":"c3b82d900ea7462521ee1dbb01a4dc826dea172ebd57209d690ad5b5bbdd7167"}
Nov 02 13:54:24.367663 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.367605    6284 scope.go:122] "RemoveContainer" containerID="8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a"
Nov 02 13:54:24.367782 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.367132    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"
Nov 02 13:54:24.371177 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.371148    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjg2x\" (UniqueName: \"kubernetes.io/projected/10c6d773-504b-4b56-ad89-5953df286c3e-kube-api-access-bjg2x\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:24.371312 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.371183    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:24.371312 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.371202    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c6d773-504b-4b56-ad89-5953df286c3e-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:24.371312 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.371216    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10c6d773-504b-4b56-ad89-5953df286c3e-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:24.395771 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.395740    6284 scope.go:122] "RemoveContainer" containerID="8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a"
Nov 02 13:54:24.396483 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:54:24.396449    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a\": container with ID starting with 8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a not found: ID does not exist" containerID="8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a"
Nov 02 13:54:24.396617 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.396498    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a"} err="failed to get container status \"8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a\": rpc error: code = NotFound desc = could not find container \"8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a\": container with ID starting with 8baf5f0bf043c63391e1c24ab49e7f50df9992c6d3b3d5a95f4bc65153836c8a not found: ID does not exist"
Nov 02 13:54:24.419764 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.419728    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"]
Nov 02 13:54:24.431017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.430974    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c148ffa-fc77c8d77-tw4qv"]
Nov 02 13:54:24.797147 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.797108    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"]
Nov 02 13:54:24.797372 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:24.797342    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr" podUID="59ba0dbe-ef60-4c3e-9153-b06987ab6213" containerName="maas-api" containerID="cri-o://fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86" gracePeriod=30
Nov 02 13:54:25.050126 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.050054    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:54:25.177775 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.177728    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-maas-api-tls\") pod \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") "
Nov 02 13:54:25.178016 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.177791    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ba0dbe-ef60-4c3e-9153-b06987ab6213-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ba0dbe-ef60-4c3e-9153-b06987ab6213-openshift-service-ca\") pod \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") "
Nov 02 13:54:25.178016 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.177828    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-metrics-certs\") pod \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") "
Nov 02 13:54:25.178016 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.177882    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ba0dbe-ef60-4c3e-9153-b06987ab6213-kube-api-access-b2drg\" (UniqueName: \"kubernetes.io/projected/59ba0dbe-ef60-4c3e-9153-b06987ab6213-kube-api-access-b2drg\") pod \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\" (UID: \"59ba0dbe-ef60-4c3e-9153-b06987ab6213\") "
Nov 02 13:54:25.178339 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.178302    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ba0dbe-ef60-4c3e-9153-b06987ab6213-openshift-service-ca" pod "59ba0dbe-ef60-4c3e-9153-b06987ab6213" (UID: "59ba0dbe-ef60-4c3e-9153-b06987ab6213"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:25.180064 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.180033    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-maas-api-tls" pod "59ba0dbe-ef60-4c3e-9153-b06987ab6213" (UID: "59ba0dbe-ef60-4c3e-9153-b06987ab6213"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:25.180171 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.180113    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ba0dbe-ef60-4c3e-9153-b06987ab6213-kube-api-access-b2drg" pod "59ba0dbe-ef60-4c3e-9153-b06987ab6213" (UID: "59ba0dbe-ef60-4c3e-9153-b06987ab6213"). InnerVolumeSpecName "kube-api-access-b2drg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:25.180171 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.180118    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-metrics-certs" pod "59ba0dbe-ef60-4c3e-9153-b06987ab6213" (UID: "59ba0dbe-ef60-4c3e-9153-b06987ab6213"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:25.279362 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.279323    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:25.279362 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.279362    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59ba0dbe-ef60-4c3e-9153-b06987ab6213-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:25.279586 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.279377    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59ba0dbe-ef60-4c3e-9153-b06987ab6213-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:25.279586 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.279390    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2drg\" (UniqueName: \"kubernetes.io/projected/59ba0dbe-ef60-4c3e-9153-b06987ab6213-kube-api-access-b2drg\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:54:25.376752 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.376645    6284 generic.go:356] "Generic (PLEG): container finished" podID="59ba0dbe-ef60-4c3e-9153-b06987ab6213" containerID="fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86" exitCode=0
Nov 02 13:54:25.376752 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.376723    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"
Nov 02 13:54:25.376752 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.376731    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr" event={"ID":"59ba0dbe-ef60-4c3e-9153-b06987ab6213","Type":"ContainerDied","Data":"fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86"}
Nov 02 13:54:25.377104 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.376808    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr" event={"ID":"59ba0dbe-ef60-4c3e-9153-b06987ab6213","Type":"ContainerDied","Data":"822d76ddb4116791a6994c34ccd94d52c7a786ed444f999e3bef888c87b3565c"}
Nov 02 13:54:25.377104 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.376830    6284 scope.go:122] "RemoveContainer" containerID="fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86"
Nov 02 13:54:25.394504 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.394478    6284 scope.go:122] "RemoveContainer" containerID="fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86"
Nov 02 13:54:25.394926 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:54:25.394901    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86\": container with ID starting with fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86 not found: ID does not exist" containerID="fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86"
Nov 02 13:54:25.395060 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.394949    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86"} err="failed to get container status \"fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86\": rpc error: code = NotFound desc = could not find container \"fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86\": container with ID starting with fc47c03a4d14902a0e66ed78e6ab24c50b736bbd04df637d6597fa896f8cfa86 not found: ID does not exist"
Nov 02 13:54:25.474477 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.474436    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"]
Nov 02 13:54:25.500833 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:25.500797    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c57ba6aa-c66c48469-gmnzr"]
Nov 02 13:54:26.170408 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:26.170375    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c6d773-504b-4b56-ad89-5953df286c3e" path="/var/lib/kubelet/pods/10c6d773-504b-4b56-ad89-5953df286c3e/volumes"
Nov 02 13:54:26.171011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:26.170991    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ba0dbe-ef60-4c3e-9153-b06987ab6213" path="/var/lib/kubelet/pods/59ba0dbe-ef60-4c3e-9153-b06987ab6213/volumes"
Nov 02 13:54:28.269133 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:28.269110    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 13:54:28.270573 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:28.270545    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 13:54:29.880654 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.880589    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"]
Nov 02 13:54:29.881063 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.881037    6284 cpu_manager.go:414] "containerMap: removing container" podUID="59ba0dbe-ef60-4c3e-9153-b06987ab6213" containerName="maas-api"
Nov 02 13:54:29.881063 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.881050    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ba0dbe-ef60-4c3e-9153-b06987ab6213" containerName="maas-api"
Nov 02 13:54:29.881063 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.881063    6284 cpu_manager.go:414] "containerMap: removing container" podUID="10c6d773-504b-4b56-ad89-5953df286c3e" containerName="maas-api"
Nov 02 13:54:29.881182 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.881068    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c6d773-504b-4b56-ad89-5953df286c3e" containerName="maas-api"
Nov 02 13:54:29.881182 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.881103    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ccb369c9-bc1b-48c7-9a18-0b021d9952b3" containerName="maas-api"
Nov 02 13:54:29.881182 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.881110    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb369c9-bc1b-48c7-9a18-0b021d9952b3" containerName="maas-api"
Nov 02 13:54:29.881182 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.881165    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb369c9-bc1b-48c7-9a18-0b021d9952b3" containerName="maas-api"
Nov 02 13:54:29.881182 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.881173    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ba0dbe-ef60-4c3e-9153-b06987ab6213" containerName="maas-api"
Nov 02 13:54:29.881182 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.881180    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c6d773-504b-4b56-ad89-5953df286c3e" containerName="maas-api"
Nov 02 13:54:29.886080 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.886054    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:29.912714 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.912651    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1a562c-serving-cert\""
Nov 02 13:54:29.915402 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:29.915369    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"]
Nov 02 13:54:30.025466 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.025419    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dlf4\" (UniqueName: \"kubernetes.io/projected/673fcfe6-3203-47b7-9578-5f6d4dd96105-kube-api-access-7dlf4\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.025697 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.025490    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-metrics-certs\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.025697 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.025647    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.025829 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.025705    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673fcfe6-3203-47b7-9578-5f6d4dd96105-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.127395 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.127350    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673fcfe6-3203-47b7-9578-5f6d4dd96105-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.127600 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.127482    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dlf4\" (UniqueName: \"kubernetes.io/projected/673fcfe6-3203-47b7-9578-5f6d4dd96105-kube-api-access-7dlf4\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.127600 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.127548    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-metrics-certs\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.127697 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.127621    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.128282 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.128257    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673fcfe6-3203-47b7-9578-5f6d4dd96105-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.130301 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.130277    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.130412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.130336    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-metrics-certs\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.163497 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.163421    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dlf4\" (UniqueName: \"kubernetes.io/projected/673fcfe6-3203-47b7-9578-5f6d4dd96105-kube-api-access-7dlf4\") pod \"maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.200512 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.200467    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:30.500491 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:30.500456    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"]
Nov 02 13:54:30.504494 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:54:30.504457    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673fcfe6_3203_47b7_9578_5f6d4dd96105.slice/crio-f39b1a7181e2abad3cba539ba98d11924ea26eaa651f2717edd6f9f65080bd26 WatchSource:0}: Error finding container f39b1a7181e2abad3cba539ba98d11924ea26eaa651f2717edd6f9f65080bd26: Status 404 returned error can't find the container with id f39b1a7181e2abad3cba539ba98d11924ea26eaa651f2717edd6f9f65080bd26
Nov 02 13:54:31.419787 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:31.419720    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447" event={"ID":"673fcfe6-3203-47b7-9578-5f6d4dd96105","Type":"ContainerStarted","Data":"2ea089f5e1f88611291d02a9edcdc99e1ec8894ca08172bd62e0d7cb3edf8723"}
Nov 02 13:54:31.420302 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:31.419827    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:31.420302 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:31.419853    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447" event={"ID":"673fcfe6-3203-47b7-9578-5f6d4dd96105","Type":"ContainerStarted","Data":"f39b1a7181e2abad3cba539ba98d11924ea26eaa651f2717edd6f9f65080bd26"}
Nov 02 13:54:31.461737 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:31.461669    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447" podStartSLOduration=2.108853002 podStartE2EDuration="2.461647831s" totalImagesPullingTime="352.794829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:31.459700717 +0000 UTC m=+603.419076324" watchObservedRunningTime="2026-11-02 13:54:31.461647831 +0000 UTC m=+603.421023439"
Nov 02 13:54:37.431696 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:37.431664    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 13:54:41.263669 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.263632    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"]
Nov 02 13:54:41.267543 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.267514    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.286907 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.286873    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ad42e0-serving-cert\""
Nov 02 13:54:41.299624 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.299584    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"]
Nov 02 13:54:41.353727 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.353684    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6qdg\" (UniqueName: \"kubernetes.io/projected/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-kube-api-access-t6qdg\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.354003 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.353762    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-metrics-certs\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.354003 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.353842    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.354003 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.353904    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.454726 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.454683    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-metrics-certs\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.454726 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.454734    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.455061 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.454856    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.455061 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.455027    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6qdg\" (UniqueName: \"kubernetes.io/projected/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-kube-api-access-t6qdg\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.455184 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:54:41.455030    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-serving-cert: secret "maas-api-e2e-worker-w4-ad42e0-serving-cert" not found
Nov 02 13:54:41.455240 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:54:41.455189    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-maas-api-tls podName:9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1 nodeName:}" failed. No retries permitted until 2026-11-02 13:54:41.955166125 +0000 UTC m=+613.914541733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-maas-api-tls") pod "maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q" (UID: "9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1") : secret "maas-api-e2e-worker-w4-ad42e0-serving-cert" not found
Nov 02 13:54:41.455553 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.455532    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.457368 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.457343    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-metrics-certs\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.476787 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.476758    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6qdg\" (UniqueName: \"kubernetes.io/projected/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-kube-api-access-t6qdg\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.960384 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.960341    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:41.963027 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:41.962989    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:42.182547 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:42.182508    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:42.486605 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:42.486564    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"]
Nov 02 13:54:42.496640 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:54:42.496603    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acc4f9b_b2cf_4f19_bdcb_9eda4bad9af1.slice/crio-fb812a8ee1b6c3434d44241e627ebbd3d70fbf24df61d21c53ad372aa1ed48b4 WatchSource:0}: Error finding container fb812a8ee1b6c3434d44241e627ebbd3d70fbf24df61d21c53ad372aa1ed48b4: Status 404 returned error can't find the container with id fb812a8ee1b6c3434d44241e627ebbd3d70fbf24df61d21c53ad372aa1ed48b4
Nov 02 13:54:43.488454 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:43.488421    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q" event={"ID":"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1","Type":"ContainerStarted","Data":"8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d"}
Nov 02 13:54:43.488878 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:43.488474    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q" event={"ID":"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1","Type":"ContainerStarted","Data":"fb812a8ee1b6c3434d44241e627ebbd3d70fbf24df61d21c53ad372aa1ed48b4"}
Nov 02 13:54:43.488878 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:43.488563    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:43.897020 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:54:43.896911    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:54:47.075299 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.075239    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q" podStartSLOduration=5.7793746089999996 podStartE2EDuration="6.075218462s" totalImagesPullingTime="295.843853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:43.533492343 +0000 UTC m=+615.492867950" watchObservedRunningTime="2026-11-02 13:54:47.075218462 +0000 UTC m=+619.034594070"
Nov 02 13:54:47.076929 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.076902    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"]
Nov 02 13:54:47.081030 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.081003    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.094166 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.094138    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-92f249-serving-cert\""
Nov 02 13:54:47.113992 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.113947    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"]
Nov 02 13:54:47.215314 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.215279    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-metrics-certs\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.215531 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.215321    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c109341-915a-457e-a0b0-e9351acad498-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.215531 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.215463    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j97r\" (UniqueName: \"kubernetes.io/projected/5c109341-915a-457e-a0b0-e9351acad498-kube-api-access-9j97r\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.215531 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.215500    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-maas-api-tls\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.316780 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.316744    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j97r\" (UniqueName: \"kubernetes.io/projected/5c109341-915a-457e-a0b0-e9351acad498-kube-api-access-9j97r\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.316780 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.316784    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-maas-api-tls\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.317105 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.316882    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-metrics-certs\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.317105 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.316911    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c109341-915a-457e-a0b0-e9351acad498-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.317627 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.317603    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c109341-915a-457e-a0b0-e9351acad498-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.319631 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.319602    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-maas-api-tls\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.319728 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.319630    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-metrics-certs\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.337836 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.337755    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j97r\" (UniqueName: \"kubernetes.io/projected/5c109341-915a-457e-a0b0-e9351acad498-kube-api-access-9j97r\") pod \"maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.395699 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.395663    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:47.715159 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:47.715057    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"]
Nov 02 13:54:47.719255 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:54:47.719217    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c109341_915a_457e_a0b0_e9351acad498.slice/crio-037d5b0a4c24598c2e6a6109682421b6ed0404a3b2bb3164d117c61a2b6974b3 WatchSource:0}: Error finding container 037d5b0a4c24598c2e6a6109682421b6ed0404a3b2bb3164d117c61a2b6974b3: Status 404 returned error can't find the container with id 037d5b0a4c24598c2e6a6109682421b6ed0404a3b2bb3164d117c61a2b6974b3
Nov 02 13:54:48.518883 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:48.518838    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs" event={"ID":"5c109341-915a-457e-a0b0-e9351acad498","Type":"ContainerStarted","Data":"5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be"}
Nov 02 13:54:48.519383 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:48.518911    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:54:48.519383 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:48.518969    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs" event={"ID":"5c109341-915a-457e-a0b0-e9351acad498","Type":"ContainerStarted","Data":"037d5b0a4c24598c2e6a6109682421b6ed0404a3b2bb3164d117c61a2b6974b3"}
Nov 02 13:54:48.557281 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:48.557215    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs" podStartSLOduration=1.160957564 podStartE2EDuration="1.55719451s" totalImagesPullingTime="396.236946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:48.556635255 +0000 UTC m=+620.516010864" watchObservedRunningTime="2026-11-02 13:54:48.55719451 +0000 UTC m=+620.516570118"
Nov 02 13:54:49.499988 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:49.499928    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 13:54:54.531716 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:54:54.531630    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 13:55:13.895231 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:55:13.895193    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:55:43.895085 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:55:43.895046    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:56:13.895190 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:56:13.895146    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:56:43.896280 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:56:43.896190    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:56:57.160270 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.160218    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"]
Nov 02 13:56:57.165186 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.165156    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.180052 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.180017    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f30173-openshift-default-dockercfg-tw6wb\""
Nov 02 13:56:57.192293 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.192252    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"]
Nov 02 13:56:57.346042 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.346003    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.346241 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.346078    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-286th\" (UniqueName: \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-kube-api-access-286th\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.346241 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.346159    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.346241 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.346212    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.346350 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.346263    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.346350 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.346326    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.346438 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.346351    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.346519 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.346495    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.346558 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.346546    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.448008 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.447873    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-286th\" (UniqueName: \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-kube-api-access-286th\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.448008 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.447931    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.448008 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.447995    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.448370 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.448047    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.448370 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.448089    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.448370 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.448123    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.448370 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.448185    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.448370 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.448236    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.448370 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.448284    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.449028 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.448653    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.449028 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.448767    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.449028 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.448973    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.449028 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.448993    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.449324 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.449190    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.451075 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.451048    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.451485 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.451462    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.469333 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.469298    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.469532 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.469510    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-286th\" (UniqueName: \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-kube-api-access-286th\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.481868 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.481842    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:57.825561 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.825526    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"]
Nov 02 13:56:57.829046 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:56:57.829013    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7b28fb_a02b_4f33_a5ce_0caf338b100c.slice/crio-d0610158797cbabe5451e98824e7acec1101f672aa94e06af85bf584098abacc WatchSource:0}: Error finding container d0610158797cbabe5451e98824e7acec1101f672aa94e06af85bf584098abacc: Status 404 returned error can't find the container with id d0610158797cbabe5451e98824e7acec1101f672aa94e06af85bf584098abacc
Nov 02 13:56:57.831426 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.831388    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:56:57.831531 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.831466    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:56:57.831531 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:57.831495    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:56:58.280299 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:58.280253    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" event={"ID":"7a7b28fb-a02b-4f33-a5ce-0caf338b100c","Type":"ContainerStarted","Data":"965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf"}
Nov 02 13:56:58.280965 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:58.280900    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" event={"ID":"7a7b28fb-a02b-4f33-a5ce-0caf338b100c","Type":"ContainerStarted","Data":"d0610158797cbabe5451e98824e7acec1101f672aa94e06af85bf584098abacc"}
Nov 02 13:56:58.331381 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:58.331317    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" podStartSLOduration=1.331300289 podStartE2EDuration="1.331300289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:58.330144785 +0000 UTC m=+750.289520394" watchObservedRunningTime="2026-11-02 13:56:58.331300289 +0000 UTC m=+750.290675899"
Nov 02 13:56:58.482673 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:58.482636    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:56:58.484281 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:58.484257    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 13:56:58.484342 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:58.484315    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 13:56:59.482574 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:59.482536    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 13:56:59.483039 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:56:59.482596    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 13:57:00.483252 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:00.483213    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 13:57:00.483775 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:00.483282    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 13:57:01.483082 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:01.483043    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 13:57:01.483277 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:01.483119    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 13:57:02.482890 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:02.482847    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 13:57:02.483199 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:02.482920    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 13:57:03.489617 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:03.489573    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 13:57:03.490164 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:03.489680    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 13:57:04.679436 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:04.679313    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:57:04.680087 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:04.679759    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:57:04.681368 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:04.681297    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 13:57:13.896872 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:57:13.896833    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:57:16.029330 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.029290    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"]
Nov 02 13:57:16.033454 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.033428    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.073788 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.073753    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"]
Nov 02 13:57:16.152154 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.152107    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.152354 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.152163    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.152354 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.152250    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.152354 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.152297    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.152463 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.152369    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz8z2\" (UniqueName: \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-kube-api-access-jz8z2\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.152463 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.152414    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.152530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.152491    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.152530 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.152517    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.152603 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.152547    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.253617 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.253570    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.254110 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.253637    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.254110 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.253726    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.254110 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.253783    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.254110 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.253815    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.254110 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.253861    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.254110 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.253912    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz8z2\" (UniqueName: \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-kube-api-access-jz8z2\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.254110 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.253999    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.254110 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.254044    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.254991 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.254475    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.254991 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.254755    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.255469 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.255442    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.255742 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.255509    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.255742 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.255676    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.257503 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.257473    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.258511 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.258487    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.276134 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.276103    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.276633 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.276610    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz8z2\" (UniqueName: \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-kube-api-access-jz8z2\") pod \"maas-default-gateway-openshift-default-6886b96f9-9zv5c\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.348847 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.348752    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:16.667786 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.667707    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"]
Nov 02 13:57:16.671012 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:57:16.670980    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c8dfe1_d27b_44d9_8c8b_757ad73ae956.slice/crio-00ec23cb9f7a55a70ea967c548310162b1bb259e5f9e260d5d2d41a69ba9086f WatchSource:0}: Error finding container 00ec23cb9f7a55a70ea967c548310162b1bb259e5f9e260d5d2d41a69ba9086f: Status 404 returned error can't find the container with id 00ec23cb9f7a55a70ea967c548310162b1bb259e5f9e260d5d2d41a69ba9086f
Nov 02 13:57:16.673485 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.673446    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:57:16.673611 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.673532    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:57:16.673611 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:16.673600    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:57:17.393846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:17.393814    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" event={"ID":"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956","Type":"ContainerStarted","Data":"4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836"}
Nov 02 13:57:17.394300 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:17.393864    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" event={"ID":"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956","Type":"ContainerStarted","Data":"00ec23cb9f7a55a70ea967c548310162b1bb259e5f9e260d5d2d41a69ba9086f"}
Nov 02 13:57:17.472959 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:17.471393    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" podStartSLOduration=1.4713578809999999 podStartE2EDuration="1.471357881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:17.446335757 +0000 UTC m=+769.405711365" watchObservedRunningTime="2026-11-02 13:57:17.471357881 +0000 UTC m=+769.430733491"
Nov 02 13:57:18.354826 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:18.354008    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:18.355251 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:18.355220    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9zv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:57:18.355485 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:18.355456    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:57:19.361411 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:19.361371    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9zv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:57:19.362631 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:19.362594    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:57:20.367964 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:20.357598    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9zv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:57:20.367964 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:20.357671    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:57:21.369960 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:21.364497    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9zv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:57:21.369960 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:21.364560    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:57:22.350456 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:22.350415    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9zv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:57:22.350768 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:22.350735    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:57:23.349345 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:23.349307    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9zv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:57:23.349771 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:23.349364    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:57:24.350109 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:24.350070    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9zv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 13:57:24.350493 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:24.350122    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 13:57:25.421465 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:25.421429    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:25.422968 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:25.422915    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:25.423805 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:25.423771    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 13:57:40.778912 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.778872    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"]
Nov 02 13:57:40.784350 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.784323    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.809350 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.809283    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lzlpd\""
Nov 02 13:57:40.809847 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.809450    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 13:57:40.810424 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.810158    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 13:57:40.810424 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.810241    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 13:57:40.810424 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.810339    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 13:57:40.810424 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.810398    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3eee531c3de6\""
Nov 02 13:57:40.814159 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.814126    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"]
Nov 02 13:57:40.863079 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.863026    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-provider-credentials\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.863288 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.863167    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-plugins-config-volume\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.863288 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.863201    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj6k8\" (UniqueName: \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-kube-api-access-nj6k8\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.863288 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.863237    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-routing-overlay-volume\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.964415 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.964350    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-provider-credentials\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.964686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.964483    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-plugins-config-volume\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.964686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.964522    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj6k8\" (UniqueName: \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-kube-api-access-nj6k8\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.964686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.964571    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-routing-overlay-volume\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.965528 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.965503    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-routing-overlay-volume\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.965662 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.965632    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-plugins-config-volume\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.968021 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.967991    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-provider-credentials\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:40.989921 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:40.989878    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj6k8\" (UniqueName: \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-kube-api-access-nj6k8\") pod \"payload-processing-external-model-68db7d7d76-mxj9z\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") " pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:41.103801 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:41.103711    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:41.423133 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:41.423048    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"]
Nov 02 13:57:41.426365 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:57:41.426333    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56ab921_c88e_4bac_ba8e_761529ac0c28.slice/crio-99831e18081bff54c17ddc044e81e40fc1d0819e7dc52f6e8695a409c3b8030c WatchSource:0}: Error finding container 99831e18081bff54c17ddc044e81e40fc1d0819e7dc52f6e8695a409c3b8030c: Status 404 returned error can't find the container with id 99831e18081bff54c17ddc044e81e40fc1d0819e7dc52f6e8695a409c3b8030c
Nov 02 13:57:41.555727 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:41.555692    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z" event={"ID":"c56ab921-c88e-4bac-ba8e-761529ac0c28","Type":"ContainerStarted","Data":"99831e18081bff54c17ddc044e81e40fc1d0819e7dc52f6e8695a409c3b8030c"}
Nov 02 13:57:43.571969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:43.571917    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z" event={"ID":"c56ab921-c88e-4bac-ba8e-761529ac0c28","Type":"ContainerStarted","Data":"a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7"}
Nov 02 13:57:43.612302 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:43.612249    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z" podStartSLOduration=1.54947704 podStartE2EDuration="3.612230943s" totalImagesPullingTime="2.062753903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:43.612194219 +0000 UTC m=+795.571569826" watchObservedRunningTime="2026-11-02 13:57:43.612230943 +0000 UTC m=+795.571606552"
Nov 02 13:57:43.897743 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:57:43.897648    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:57:44.578507 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.578475    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:44.690690 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.690655    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"]
Nov 02 13:57:44.694388 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.694368    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.712411 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.712383    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"default-dockercfg-clf8h\""
Nov 02 13:57:44.731020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.730988    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"openshift-service-ca.crt\""
Nov 02 13:57:44.744247 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.744210    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"]
Nov 02 13:57:44.808846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.808800    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-dshm\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.809124 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.808868    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f52e51f3-ba95-45c4-a445-98bb17486ed7-tls-certs\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.809124 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.808971    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-model-cache\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.809124 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.809016    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458h4\" (UniqueName: \"kubernetes.io/projected/f52e51f3-ba95-45c4-a445-98bb17486ed7-kube-api-access-458h4\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.809124 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.809041    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-home\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.809294 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.809202    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-tmp-dir\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.812395 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.812363    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"e2e-conflict-a-409544bc-kserve-self-signed-certs\""
Nov 02 13:57:44.835801 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.835714    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"kube-root-ca.crt\""
Nov 02 13:57:44.910411 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.910355    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-dshm\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.910603 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.910523    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f52e51f3-ba95-45c4-a445-98bb17486ed7-tls-certs\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.910603 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.910566    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-model-cache\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.910746 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.910605    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458h4\" (UniqueName: \"kubernetes.io/projected/f52e51f3-ba95-45c4-a445-98bb17486ed7-kube-api-access-458h4\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.910746 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.910623    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-home\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.910746 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.910722    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-tmp-dir\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.911283 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.911014    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-model-cache\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.911283 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.911034    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-tmp-dir\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.911283 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.911235    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-home\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.922650 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.922615    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f52e51f3-ba95-45c4-a445-98bb17486ed7-tls-certs\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.922886 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.922866    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-dshm\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:44.936028 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:44.935994    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458h4\" (UniqueName: \"kubernetes.io/projected/f52e51f3-ba95-45c4-a445-98bb17486ed7-kube-api-access-458h4\") pod \"e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:45.041812 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:45.041768    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:45.380094 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:45.380008    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"]
Nov 02 13:57:45.384347 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:57:45.384310    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf52e51f3_ba95_45c4_a445_98bb17486ed7.slice/crio-5a48999d5e3916d0f816b931eb18769e4feb194de72f74faa038e5e9a18e84eb WatchSource:0}: Error finding container 5a48999d5e3916d0f816b931eb18769e4feb194de72f74faa038e5e9a18e84eb: Status 404 returned error can't find the container with id 5a48999d5e3916d0f816b931eb18769e4feb194de72f74faa038e5e9a18e84eb
Nov 02 13:57:45.585412 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:45.585370    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5" event={"ID":"f52e51f3-ba95-45c4-a445-98bb17486ed7","Type":"ContainerStarted","Data":"5a48999d5e3916d0f816b931eb18769e4feb194de72f74faa038e5e9a18e84eb"}
Nov 02 13:57:48.655254 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:48.655206    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5" event={"ID":"f52e51f3-ba95-45c4-a445-98bb17486ed7","Type":"ContainerStarted","Data":"a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904"}
Nov 02 13:57:48.729179 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:48.729118    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5" podStartSLOduration=2.730123444 podStartE2EDuration="4.729100875s" totalImagesPullingTime="1.998977431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:48.727066183 +0000 UTC m=+800.686441791" watchObservedRunningTime="2026-11-02 13:57:48.729100875 +0000 UTC m=+800.688476481"
Nov 02 13:57:49.042362 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:49.042321    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:55.042997 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:55.042884    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:55.055977 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:55.055929    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:57:55.589267 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:55.589239    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:57:55.712823 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:57:55.712792    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:58:04.654701 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.654589    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"]
Nov 02 13:58:04.660435 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.660408    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.686488 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.686442    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"openshift-service-ca.crt\""
Nov 02 13:58:04.686748 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.686581    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"]
Nov 02 13:58:04.686748 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.686726    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"distinct-w2-0969b1-kserve-self-signed-certs\""
Nov 02 13:58:04.687645 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.686849    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"kube-root-ca.crt\""
Nov 02 13:58:04.687645 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.686915    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"default-dockercfg-9z88n\""
Nov 02 13:58:04.771354 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.771307    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzld\" (UniqueName: \"kubernetes.io/projected/b3c53689-f9e1-42f5-b613-27c0f318f778-kube-api-access-nxzld\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.771571 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.771382    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-model-cache\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.771571 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.771430    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-tmp-dir\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.771571 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.771483    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-dshm\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.771773 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.771609    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c53689-f9e1-42f5-b613-27c0f318f778-tls-certs\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.771773 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.771640    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-home\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.875959 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.872535    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-tmp-dir\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.875959 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.872623    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-dshm\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.875959 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.872760    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c53689-f9e1-42f5-b613-27c0f318f778-tls-certs\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.875959 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.872796    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-home\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.875959 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.872845    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzld\" (UniqueName: \"kubernetes.io/projected/b3c53689-f9e1-42f5-b613-27c0f318f778-kube-api-access-nxzld\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.875959 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.872896    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-model-cache\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.875959 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.873432    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-model-cache\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.875959 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.873705    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-tmp-dir\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.875959 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.875094    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-home\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.883674 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.883634    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-dshm\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.903171 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.903137    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzld\" (UniqueName: \"kubernetes.io/projected/b3c53689-f9e1-42f5-b613-27c0f318f778-kube-api-access-nxzld\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.922740 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.922616    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c53689-f9e1-42f5-b613-27c0f318f778-tls-certs\") pod \"distinct-w2-0969b1-kserve-7c99747844-2hfmj\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") " pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:04.988471 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:04.988154    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:05.522814 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:58:05.522746    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3c53689_f9e1_42f5_b613_27c0f318f778.slice/crio-8d5335571c625f32be02002f6b8648ba08a60b7c4a1364163c07f3f31abdf8a0 WatchSource:0}: Error finding container 8d5335571c625f32be02002f6b8648ba08a60b7c4a1364163c07f3f31abdf8a0: Status 404 returned error can't find the container with id 8d5335571c625f32be02002f6b8648ba08a60b7c4a1364163c07f3f31abdf8a0
Nov 02 13:58:05.525698 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:05.525662    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"]
Nov 02 13:58:05.797458 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:05.793252    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" event={"ID":"b3c53689-f9e1-42f5-b613-27c0f318f778","Type":"ContainerStarted","Data":"9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c"}
Nov 02 13:58:05.797458 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:05.793310    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" event={"ID":"b3c53689-f9e1-42f5-b613-27c0f318f778","Type":"ContainerStarted","Data":"8d5335571c625f32be02002f6b8648ba08a60b7c4a1364163c07f3f31abdf8a0"}
Nov 02 13:58:05.837993 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:05.837907    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podStartSLOduration=1.8378807830000001 podStartE2EDuration="1.837880783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:05.837167543 +0000 UTC m=+817.796543153" watchObservedRunningTime="2026-11-02 13:58:05.837880783 +0000 UTC m=+817.797256390"
Nov 02 13:58:05.989961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:05.989035    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:09.780795 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.780755    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"]
Nov 02 13:58:09.785395 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.785368    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.800189 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.800163    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1a562c-openshift-default-dockercfg-smz6x\""
Nov 02 13:58:09.812607 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.812567    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"]
Nov 02 13:58:09.881592 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.881546    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21504bd-e77a-403a-ad00-7331870c5b59-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.881592 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.881594    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.881828 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.881630    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.881828 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.881726    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.881828 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.881789    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.881973 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.881854    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.881973 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.881890    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.881973 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.881925    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zsps\" (UniqueName: \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-kube-api-access-7zsps\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.882083 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.882069    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21504bd-e77a-403a-ad00-7331870c5b59-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.982970 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.982892    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21504bd-e77a-403a-ad00-7331870c5b59-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983165 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983005    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21504bd-e77a-403a-ad00-7331870c5b59-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983165 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983046    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983165 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983097    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983165 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983124    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983378 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983188    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983378 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983225    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983378 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983260    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983378 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983297    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zsps\" (UniqueName: \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-kube-api-access-7zsps\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983587 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983517    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983648 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983631    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983699 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983666    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.983866 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983839    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21504bd-e77a-403a-ad00-7331870c5b59-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.984027 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.983876    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.985712 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.985692    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:09.985857 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:09.985834    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21504bd-e77a-403a-ad00-7331870c5b59-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:10.005108 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.005067    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:10.005271 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.005126    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zsps\" (UniqueName: \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-kube-api-access-7zsps\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:10.103485 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.103392    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:10.429495 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.429401    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"]
Nov 02 13:58:10.433443 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:58:10.433405    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21504bd_e77a_403a_ad00_7331870c5b59.slice/crio-2e30c7ca53e149b9acec3c30d1c65040bffc493393d577fc073fbfa8d8ee4f96 WatchSource:0}: Error finding container 2e30c7ca53e149b9acec3c30d1c65040bffc493393d577fc073fbfa8d8ee4f96: Status 404 returned error can't find the container with id 2e30c7ca53e149b9acec3c30d1c65040bffc493393d577fc073fbfa8d8ee4f96
Nov 02 13:58:10.436197 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.436157    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:10.436322 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.436250    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:10.436322 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.436297    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:10.553302 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.549971    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"]
Nov 02 13:58:10.826291 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.826256    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" event={"ID":"b21504bd-e77a-403a-ad00-7331870c5b59","Type":"ContainerStarted","Data":"fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6"}
Nov 02 13:58:10.826690 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.826319    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" event={"ID":"b21504bd-e77a-403a-ad00-7331870c5b59","Type":"ContainerStarted","Data":"2e30c7ca53e149b9acec3c30d1c65040bffc493393d577fc073fbfa8d8ee4f96"}
Nov 02 13:58:10.868696 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:10.868633    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podStartSLOduration=1.8686180019999998 podStartE2EDuration="1.868618002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:10.868005789 +0000 UTC m=+822.827381408" watchObservedRunningTime="2026-11-02 13:58:10.868618002 +0000 UTC m=+822.827993673"
Nov 02 13:58:11.104329 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:11.104234    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:11.105845 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:11.105816    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:58:11.106035 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:11.105870    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 13:58:12.104791 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:12.104753    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:58:12.105256 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:12.104810    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 13:58:13.104571 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:13.104532    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:58:13.104916 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:13.104884    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 13:58:13.901404 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:58:13.901360    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:58:14.104415 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:14.104376    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:58:14.104602 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:14.104440    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 13:58:14.989550 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:14.989509    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:15.004591 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.004553    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:15.104704 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.104658    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:58:15.104889 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.104721    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 13:58:15.555877 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.553015    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z" podUID="c56ab921-c88e-4bac-ba8e-761529ac0c28" containerName="payload-processing" containerID="cri-o://a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7" gracePeriod=25
Nov 02 13:58:15.586896 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:58:15.586870    6284 logging.go:55] [core] [Channel #26 SubChannel #27]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.53:50052", ServerName: "10.134.0.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.53:50052: connect: connection refused"
Nov 02 13:58:15.847894 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.847862    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:58:15.885683 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.885624    6284 generic.go:356] "Generic (PLEG): container finished" podID="c56ab921-c88e-4bac-ba8e-761529ac0c28" containerID="a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7" exitCode=0
Nov 02 13:58:15.886715 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.886609    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"
Nov 02 13:58:15.886863 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.886654    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z" event={"ID":"c56ab921-c88e-4bac-ba8e-761529ac0c28","Type":"ContainerDied","Data":"a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7"}
Nov 02 13:58:15.886863 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.886826    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z" event={"ID":"c56ab921-c88e-4bac-ba8e-761529ac0c28","Type":"ContainerDied","Data":"99831e18081bff54c17ddc044e81e40fc1d0819e7dc52f6e8695a409c3b8030c"}
Nov 02 13:58:15.886981 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.886863    6284 scope.go:122] "RemoveContainer" containerID="a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7"
Nov 02 13:58:15.897144 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.897117    6284 scope.go:122] "RemoveContainer" containerID="a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7"
Nov 02 13:58:15.897586 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:58:15.897557    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7\": container with ID starting with a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7 not found: ID does not exist" containerID="a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7"
Nov 02 13:58:15.897735 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.897597    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7"} err="failed to get container status \"a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7\": rpc error: code = NotFound desc = could not find container \"a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7\": container with ID starting with a891c8a694892249202dd446c8d28481230fe6cf1ad8158b17c2714d5e8a2bc7 not found: ID does not exist"
Nov 02 13:58:15.900112 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.900083    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 13:58:15.976964 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.976520    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-routing-overlay-volume\") pod \"c56ab921-c88e-4bac-ba8e-761529ac0c28\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") "
Nov 02 13:58:15.976964 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.976664    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-plugins-config-volume\") pod \"c56ab921-c88e-4bac-ba8e-761529ac0c28\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") "
Nov 02 13:58:15.976964 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.976746    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-provider-credentials\") pod \"c56ab921-c88e-4bac-ba8e-761529ac0c28\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") "
Nov 02 13:58:15.976964 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.976852    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-kube-api-access-nj6k8\" (UniqueName: \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-kube-api-access-nj6k8\") pod \"c56ab921-c88e-4bac-ba8e-761529ac0c28\" (UID: \"c56ab921-c88e-4bac-ba8e-761529ac0c28\") "
Nov 02 13:58:15.978998 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.978820    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-plugins-config-volume" pod "c56ab921-c88e-4bac-ba8e-761529ac0c28" (UID: "c56ab921-c88e-4bac-ba8e-761529ac0c28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:15.979171 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.979034    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-routing-overlay-volume" pod "c56ab921-c88e-4bac-ba8e-761529ac0c28" (UID: "c56ab921-c88e-4bac-ba8e-761529ac0c28"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:15.982190 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.982153    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-provider-credentials" pod "c56ab921-c88e-4bac-ba8e-761529ac0c28" (UID: "c56ab921-c88e-4bac-ba8e-761529ac0c28"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:15.984060 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:15.984027    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-kube-api-access-nj6k8" pod "c56ab921-c88e-4bac-ba8e-761529ac0c28" (UID: "c56ab921-c88e-4bac-ba8e-761529ac0c28"). InnerVolumeSpecName "kube-api-access-nj6k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:16.078259 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:16.078165    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:58:16.078259 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:16.078203    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj6k8\" (UniqueName: \"kubernetes.io/projected/c56ab921-c88e-4bac-ba8e-761529ac0c28-kube-api-access-nj6k8\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:58:16.078259 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:16.078216    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:58:16.078259 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:16.078229    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56ab921-c88e-4bac-ba8e-761529ac0c28-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:58:16.104707 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:16.104671    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:58:16.104886 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:16.104735    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 13:58:16.221856 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:16.221807    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"]
Nov 02 13:58:16.225978 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:16.225928    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z"]
Nov 02 13:58:16.587048 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:16.587001    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-68db7d7d76-mxj9z" podUID="c56ab921-c88e-4bac-ba8e-761529ac0c28" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.53:50052\" within 1s: context deadline exceeded"
Nov 02 13:58:17.104448 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:17.104326    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:58:17.104448 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:17.104393    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 13:58:18.109136 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:18.109098    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:18.109529 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:18.109483    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:18.110248 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:18.110227    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 13:58:18.169033 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:18.168996    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56ab921-c88e-4bac-ba8e-761529ac0c28" path="/var/lib/kubelet/pods/c56ab921-c88e-4bac-ba8e-761529ac0c28/volumes"
Nov 02 13:58:19.066888 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.066849    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"]
Nov 02 13:58:19.067707 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.067676    6284 cpu_manager.go:414] "containerMap: removing container" podUID="c56ab921-c88e-4bac-ba8e-761529ac0c28" containerName="payload-processing"
Nov 02 13:58:19.067707 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.067708    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56ab921-c88e-4bac-ba8e-761529ac0c28" containerName="payload-processing"
Nov 02 13:58:19.067882 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.067848    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56ab921-c88e-4bac-ba8e-761529ac0c28" containerName="payload-processing"
Nov 02 13:58:19.073056 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.073032    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.090762 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.090725    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"]
Nov 02 13:58:19.096376 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.096349    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"e2e-conflict-b-409544bc-kserve-self-signed-certs\""
Nov 02 13:58:19.213782 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.213730    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-tmp-dir\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.214343 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.213800    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/412c0051-d33b-487c-a56a-fca13299fa75-tls-certs\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.214343 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.213911    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-dshm\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.214343 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.213999    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7f9\" (UniqueName: \"kubernetes.io/projected/412c0051-d33b-487c-a56a-fca13299fa75-kube-api-access-gr7f9\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.214343 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.214085    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-home\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.214343 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.214116    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-model-cache\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.315480 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.315440    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-home\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.315669 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.315499    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-model-cache\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.315669 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.315559    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-tmp-dir\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.315669 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.315631    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/412c0051-d33b-487c-a56a-fca13299fa75-tls-certs\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.315825 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.315675    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-dshm\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.315887 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.315872    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7f9\" (UniqueName: \"kubernetes.io/projected/412c0051-d33b-487c-a56a-fca13299fa75-kube-api-access-gr7f9\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.315988 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.315921    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-home\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.316386 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.316360    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-tmp-dir\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.316482 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.316393    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-model-cache\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.318165 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.318100    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-dshm\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.318440 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.318425    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/412c0051-d33b-487c-a56a-fca13299fa75-tls-certs\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.336858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.336826    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7f9\" (UniqueName: \"kubernetes.io/projected/412c0051-d33b-487c-a56a-fca13299fa75-kube-api-access-gr7f9\") pod \"e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") " pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.387407 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.387367    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:19.689699 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.689609    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"]
Nov 02 13:58:19.693098 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:58:19.693060    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod412c0051_d33b_487c_a56a_fca13299fa75.slice/crio-0b0432d0030b0543c2931ad05a8b18adc06356d0fc444462bd4115e05c5d2f73 WatchSource:0}: Error finding container 0b0432d0030b0543c2931ad05a8b18adc06356d0fc444462bd4115e05c5d2f73: Status 404 returned error can't find the container with id 0b0432d0030b0543c2931ad05a8b18adc06356d0fc444462bd4115e05c5d2f73
Nov 02 13:58:19.915774 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.915735    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m" event={"ID":"412c0051-d33b-487c-a56a-fca13299fa75","Type":"ContainerStarted","Data":"077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe"}
Nov 02 13:58:19.915774 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.915781    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m" event={"ID":"412c0051-d33b-487c-a56a-fca13299fa75","Type":"ContainerStarted","Data":"0b0432d0030b0543c2931ad05a8b18adc06356d0fc444462bd4115e05c5d2f73"}
Nov 02 13:58:19.962856 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:19.955149    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m" podStartSLOduration=0.955128382 podStartE2EDuration="955.128382ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:19.951818184 +0000 UTC m=+831.911193794" watchObservedRunningTime="2026-11-02 13:58:19.955128382 +0000 UTC m=+831.914503990"
Nov 02 13:58:20.388125 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:20.388088    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:27.445789 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.445750    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"]
Nov 02 13:58:27.449780 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.449750    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.478419 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.478379    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"]
Nov 02 13:58:27.606565 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.606522    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.606802 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.606587    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.606802 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.606770    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.606989 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.606816    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkdm2\" (UniqueName: \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-kube-api-access-pkdm2\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.606989 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.606885    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.607113 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.606975    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.607113 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.607026    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.607113 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.607069    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.607113 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.607097    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.708275 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.708173    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.708275 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.708256    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.708519 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.708341    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.708519 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.708370    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkdm2\" (UniqueName: \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-kube-api-access-pkdm2\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.708519 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.708404    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.708519 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.708437    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.708519 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.708491    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.708837 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.708811    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.708908 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.708611    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.708908 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.708897    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.709057 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.709033    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.709122 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.709067    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.709122 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.709033    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.709236 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.709195    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.711082 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.711057    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.711282 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.711263    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.730415 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.730376    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.730597 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.730486    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkdm2\" (UniqueName: \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-kube-api-access-pkdm2\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:27.765246 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:27.765202    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:28.085926 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:28.085878    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"]
Nov 02 13:58:28.091099 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:58:28.091066    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8c4e7e_20e8_42e6_92f4_4c105e41f52e.slice/crio-99f4907fd6fbed5785c7b5fcce2027cfeefdc7c4e6ec1058057abc9b1b3dd919 WatchSource:0}: Error finding container 99f4907fd6fbed5785c7b5fcce2027cfeefdc7c4e6ec1058057abc9b1b3dd919: Status 404 returned error can't find the container with id 99f4907fd6fbed5785c7b5fcce2027cfeefdc7c4e6ec1058057abc9b1b3dd919
Nov 02 13:58:28.093585 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:28.093549    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:28.093685 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:28.093640    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:28.093685 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:28.093673    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:28.981185 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:28.981137    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" event={"ID":"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e","Type":"ContainerStarted","Data":"477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906"}
Nov 02 13:58:28.981742 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:28.981220    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" event={"ID":"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e","Type":"ContainerStarted","Data":"99f4907fd6fbed5785c7b5fcce2027cfeefdc7c4e6ec1058057abc9b1b3dd919"}
Nov 02 13:58:29.048561 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:29.038210    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" podStartSLOduration=2.03818924 podStartE2EDuration="2.03818924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:29.033408533 +0000 UTC m=+840.992784178" watchObservedRunningTime="2026-11-02 13:58:29.03818924 +0000 UTC m=+840.997564848"
Nov 02 13:58:29.388340 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:29.388302    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:29.401573 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:29.401541    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:29.774210 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:29.774172    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:29.775918 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:29.775654    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 13:58:29.775918 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:29.775719    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 13:58:29.999102 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:29.999067    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:58:30.768962 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:30.768400    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 13:58:30.768962 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:30.768469    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 13:58:31.769960 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:31.769419    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 13:58:31.769960 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:31.769490    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 13:58:32.769878 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:32.769407    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 13:58:32.769878 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:32.769477    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 13:58:33.767475 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:33.767434    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 13:58:33.767773 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:33.767725    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 13:58:34.766029 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:34.765991    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 13:58:34.766540 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:34.766061    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 13:58:35.771046 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:35.771013    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:35.771472 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:35.771452    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:35.772353 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:35.772330    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 13:58:43.625558 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:43.625474    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hwr8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:43.626478 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:43.626428    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:43.945570 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:58:43.945477    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:58:49.580543 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.580499    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"]
Nov 02 13:58:49.585600 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.585568    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.616040 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.616010    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-92f249-openshift-default-dockercfg-p2t2n\""
Nov 02 13:58:49.619302 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.619266    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"]
Nov 02 13:58:49.744654 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.744611    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.744891 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.744680    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.744891 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.744725    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.745047 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.744888    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.745047 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.744952    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.745047 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.745037    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.745207 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.745102    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85w8j\" (UniqueName: \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-kube-api-access-85w8j\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.745207 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.745138    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.745311 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.745239    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.846374 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.846277    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.846374 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.846340    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85w8j\" (UniqueName: \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-kube-api-access-85w8j\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.846640 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.846462    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.846640 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.846531    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.846640 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.846611    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.846799 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.846668    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.846799 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.846696    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.846799 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.846774    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.847014 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.846812    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.847111 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.847006    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.847187 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.847158    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.847384 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.847353    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.847510 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.847445    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.847510 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.847462    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.849144 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.849120    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.849446 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.849428    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.869258 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.869224    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.869418 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.869277    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85w8j\" (UniqueName: \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-kube-api-access-85w8j\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:49.901258 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:49.901216    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:50.236556 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:50.236518    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"]
Nov 02 13:58:50.238902 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:58:50.238866    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5985c62f_afea_4cd9_b2c1_7ebc41f24c5f.slice/crio-9340062ea8fd726ce973ecd6ef059a752c480f4231cf84499bf71f6a9db0c1d5 WatchSource:0}: Error finding container 9340062ea8fd726ce973ecd6ef059a752c480f4231cf84499bf71f6a9db0c1d5: Status 404 returned error can't find the container with id 9340062ea8fd726ce973ecd6ef059a752c480f4231cf84499bf71f6a9db0c1d5
Nov 02 13:58:50.242472 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:50.242419    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:50.242631 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:50.242521    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:50.242631 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:50.242565    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:51.223040 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:51.222997    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" event={"ID":"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f","Type":"ContainerStarted","Data":"ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b"}
Nov 02 13:58:51.223570 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:51.223070    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" event={"ID":"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f","Type":"ContainerStarted","Data":"9340062ea8fd726ce973ecd6ef059a752c480f4231cf84499bf71f6a9db0c1d5"}
Nov 02 13:58:51.300021 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:51.295793    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" podStartSLOduration=2.29576835 podStartE2EDuration="2.29576835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:51.281559326 +0000 UTC m=+863.240934933" watchObservedRunningTime="2026-11-02 13:58:51.29576835 +0000 UTC m=+863.255143956"
Nov 02 13:58:51.905090 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:51.905052    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:51.905504 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:51.905478    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 13:58:51.905686 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:51.905662    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 13:58:52.902971 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:52.902523    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 13:58:52.902971 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:52.902607    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 13:58:53.902217 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:53.902173    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 13:58:53.902473 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:53.902238    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 13:58:54.901651 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:54.901611    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 13:58:54.902112 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:54.901683    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 13:58:54.977924 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:54.977874    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"]
Nov 02 13:58:54.984669 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:54.984627    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.024539 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.024488    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.024744 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.024564    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.024744 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.024586    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.024744 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.024609    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.024744 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.024687    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.024744 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.024721    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.025054 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.024753    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.025054 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.024812    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.025054 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.024846    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5ws\" (UniqueName: \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-kube-api-access-pz5ws\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.028021 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.027964    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"]
Nov 02 13:58:55.126102 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126060    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126330 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126116    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126330 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126226    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126330 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126289    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126330 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126317    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5ws\" (UniqueName: \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-kube-api-access-pz5ws\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126548 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126342    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126548 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126411    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126548 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126436    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126548 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126513    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126752 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126595    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126752 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126670    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126883 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126860    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.126982 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.126930    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.127299 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.127281    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.128805 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.128773    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.129044 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.129024    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.148196 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.148157    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5ws\" (UniqueName: \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-kube-api-access-pz5ws\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.174021 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.169677    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.310156 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.310114    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:55.678960 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:58:55.678876    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0db0e10_3bf4_46f5_be72_2ec3687ffcfa.slice/crio-96af5885e6420d050a2d2d6c97b319e3a57d5c663b6476d7db61c82fafbaf034 WatchSource:0}: Error finding container 96af5885e6420d050a2d2d6c97b319e3a57d5c663b6476d7db61c82fafbaf034: Status 404 returned error can't find the container with id 96af5885e6420d050a2d2d6c97b319e3a57d5c663b6476d7db61c82fafbaf034
Nov 02 13:58:55.683056 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.683021    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:55.683206 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.683106    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:55.683206 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.683152    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:58:55.683963 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.683902    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"]
Nov 02 13:58:55.902448 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.902414    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 13:58:55.902992 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:55.902958    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 13:58:56.308357 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:56.308319    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" event={"ID":"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa","Type":"ContainerStarted","Data":"19e67c7b68465aea7d9b7162a8d3672e86788e55534055eb64802662c5ce4a58"}
Nov 02 13:58:56.308558 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:56.308375    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" event={"ID":"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa","Type":"ContainerStarted","Data":"96af5885e6420d050a2d2d6c97b319e3a57d5c663b6476d7db61c82fafbaf034"}
Nov 02 13:58:56.310985 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:56.310782    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:58:56.312262 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:56.312239    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 13:58:56.312362 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:56.312289    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 13:58:56.377961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:56.373061    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" podStartSLOduration=2.373037904 podStartE2EDuration="2.373037904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:56.369486841 +0000 UTC m=+868.328862448" watchObservedRunningTime="2026-11-02 13:58:56.373037904 +0000 UTC m=+868.332413511"
Nov 02 13:58:56.902322 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:56.902281    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Nov 02 13:58:56.902502 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:56.902368    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Nov 02 13:58:57.310662 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:57.310619    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 13:58:57.310834 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:57.310691    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 13:58:57.906677 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:57.906634    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:57.907129 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:57.907051    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:57.907824 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:57.907806    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 13:58:58.310918 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:58.310879    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 13:58:58.311121 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:58.310964    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 13:58:59.311258 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:59.311211    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 13:58:59.311773 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:58:59.311278    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 13:59:00.311260 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:00.311217    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 13:59:00.311696 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:00.311275    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 13:59:01.310811 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:01.310764    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 13:59:01.311059 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:01.310840    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 13:59:02.311233 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:02.311195    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 13:59:02.311664 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:02.311261    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 13:59:03.317104 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:03.317068    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:59:03.317604 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:03.317370    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:59:03.318410 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:03.318388    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 13:59:09.977073 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:09.977031    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"]
Nov 02 13:59:09.981316 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:09.981292    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.019157 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.019113    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"]
Nov 02 13:59:10.041656 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.041620    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4t6\" (UniqueName: \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-kube-api-access-6c4t6\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.041850 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.041677    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.041850 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.041796    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.042007 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.041900    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.042007 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.041956    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.042007 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.041987    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.042180 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.042155    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.042326 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.042310    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.042409 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.042396    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.143760 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.143716    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.143760 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.143762    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4t6\" (UniqueName: \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-kube-api-access-6c4t6\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144107 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.143805    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144107 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.143846    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144107 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.143876    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144107 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.143895    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144107 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.143918    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144107 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.144079    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144399 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.144217    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144399 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.144226    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144399 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.144245    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.144509    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144619 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.144600    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.144839 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.144818    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.146272 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.146245    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.146853 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.146829    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.165783 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.165753    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4t6\" (UniqueName: \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-kube-api-access-6c4t6\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.165783 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.165769    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.297182 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.297143    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:10.624227 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.624193    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"]
Nov 02 13:59:10.626655 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:59:10.626537    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6abbb9f_1bcc_4ec1_bff5_6a727c898e4f.slice/crio-6ad31bb8bbaaa4eaf495334b686141f0d3c5aa2d317973e4477b426a4d2b78d6 WatchSource:0}: Error finding container 6ad31bb8bbaaa4eaf495334b686141f0d3c5aa2d317973e4477b426a4d2b78d6: Status 404 returned error can't find the container with id 6ad31bb8bbaaa4eaf495334b686141f0d3c5aa2d317973e4477b426a4d2b78d6
Nov 02 13:59:10.629158 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.629121    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:10.629274 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.629195    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:10.629274 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:10.629224    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:11.408415 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:11.408374    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" event={"ID":"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f","Type":"ContainerStarted","Data":"37d2c6524eb0f6a8bd23d9a129cd9c1d7cb6062a82f72a0092bdb626357a8f24"}
Nov 02 13:59:11.408846 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:11.408441    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" event={"ID":"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f","Type":"ContainerStarted","Data":"6ad31bb8bbaaa4eaf495334b686141f0d3c5aa2d317973e4477b426a4d2b78d6"}
Nov 02 13:59:12.297699 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.297655    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:12.299612 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.299581    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:59:12.299752 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.299655    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 13:59:12.689226 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.689107    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" podStartSLOduration=3.689089924 podStartE2EDuration="3.689089924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:11.470674165 +0000 UTC m=+883.430049772" watchObservedRunningTime="2026-11-02 13:59:12.689089924 +0000 UTC m=+884.648465534"
Nov 02 13:59:12.689655 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.689421    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"]
Nov 02 13:59:12.693616 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.693583    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.720441 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.720406    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"]
Nov 02 13:59:12.794660 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.794618    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.794865 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.794738    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.794865 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.794803    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.794865 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.794827    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hfkv\" (UniqueName: \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-kube-api-access-4hfkv\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.795017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.794967    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.795017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.795007    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.795082 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.795064    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.795114 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.795092    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907e32be-943b-47c5-8091-2bc94b31f433-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.795157 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.795119    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907e32be-943b-47c5-8091-2bc94b31f433-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896190 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896140    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896411 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896238    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896411 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896291    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896411 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896323    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hfkv\" (UniqueName: \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-kube-api-access-4hfkv\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896411 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896378    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896617 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896420    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896617 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896463    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896617 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896496    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907e32be-943b-47c5-8091-2bc94b31f433-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896617 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896536    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907e32be-943b-47c5-8091-2bc94b31f433-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896812 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896640    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896812 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896675    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.896812 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.896791    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.897038 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.897014    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.897227 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.897208    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907e32be-943b-47c5-8091-2bc94b31f433-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.898764 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.898742    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.899113 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.899087    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907e32be-943b-47c5-8091-2bc94b31f433-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.919270 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.919231    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:12.919444 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:12.919327    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hfkv\" (UniqueName: \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-kube-api-access-4hfkv\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:13.008205 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.008167    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"]
Nov 02 13:59:13.012842 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.012785    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:13.013467 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.013446    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.029628 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.029599    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-0969b1-openshift-default-dockercfg-8rlbj\""
Nov 02 13:59:13.041780 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.041746    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"]
Nov 02 13:59:13.101402 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.100497    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.101584 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.101547    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112fb288-71fc-4554-a30d-2f7776fb51ea-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.102982 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.101699    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.102982 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.101745    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.102982 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.101794    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.102982 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.101970    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.102982 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.102035    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhhm4\" (UniqueName: \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-kube-api-access-bhhm4\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.102982 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.102074    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.102982 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.102177    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.203242 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.203198    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhhm4\" (UniqueName: \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-kube-api-access-bhhm4\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.203242 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.203253    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.203523 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.203286    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.203523 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.203381    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.203523 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.203405    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112fb288-71fc-4554-a30d-2f7776fb51ea-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.203523 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.203446    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.203523 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.203482    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.203523 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.203511    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.203818 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.203554    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.204090 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.204017    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.204090 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.204016    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.204304 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.204102    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.204304 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.204210    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.204419 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.204345    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112fb288-71fc-4554-a30d-2f7776fb51ea-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.205821 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.205800    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.205994 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.205977    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.225054 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.225015    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.225220 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.225059    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhhm4\" (UniqueName: \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-kube-api-access-bhhm4\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.298293 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.298189    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:59:13.298293 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.298255    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 13:59:13.344259 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.344215    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:13.392018 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.391979    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"]
Nov 02 13:59:13.394052 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:59:13.393986    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907e32be_943b_47c5_8091_2bc94b31f433.slice/crio-1617d3e5ecbb7394234af84ddba08036a25345d60262606aaea0d07eb1a68368 WatchSource:0}: Error finding container 1617d3e5ecbb7394234af84ddba08036a25345d60262606aaea0d07eb1a68368: Status 404 returned error can't find the container with id 1617d3e5ecbb7394234af84ddba08036a25345d60262606aaea0d07eb1a68368
Nov 02 13:59:13.397003 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.396967    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:13.397082 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.397045    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:13.397129 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.397079    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:13.424542 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.424505    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" event={"ID":"907e32be-943b-47c5-8091-2bc94b31f433","Type":"ContainerStarted","Data":"1617d3e5ecbb7394234af84ddba08036a25345d60262606aaea0d07eb1a68368"}
Nov 02 13:59:13.727312 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.727280    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"]
Nov 02 13:59:13.728681 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:59:13.728645    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112fb288_71fc_4554_a30d_2f7776fb51ea.slice/crio-bfaafa95286166a49f214cd160f81abd4ea8d0441215a154d3debbf4c9ad190c WatchSource:0}: Error finding container bfaafa95286166a49f214cd160f81abd4ea8d0441215a154d3debbf4c9ad190c: Status 404 returned error can't find the container with id bfaafa95286166a49f214cd160f81abd4ea8d0441215a154d3debbf4c9ad190c
Nov 02 13:59:13.731400 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.731368    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:13.731497 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.731437    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:13.731497 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.731470    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:13.987957 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:13.948538    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"]
Nov 02 13:59:14.129853 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:59:14.096616    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:59:14.298426 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:14.298379    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:59:14.298655 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:14.298441    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 13:59:14.568960 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:14.568688    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" event={"ID":"907e32be-943b-47c5-8091-2bc94b31f433","Type":"ContainerStarted","Data":"f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48"}
Nov 02 13:59:14.602993 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:14.588364    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" event={"ID":"112fb288-71fc-4554-a30d-2f7776fb51ea","Type":"ContainerStarted","Data":"9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8"}
Nov 02 13:59:14.602993 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:14.589724    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" event={"ID":"112fb288-71fc-4554-a30d-2f7776fb51ea","Type":"ContainerStarted","Data":"bfaafa95286166a49f214cd160f81abd4ea8d0441215a154d3debbf4c9ad190c"}
Nov 02 13:59:14.637961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:14.629377    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" podStartSLOduration=2.629354465 podStartE2EDuration="2.629354465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:14.62768679 +0000 UTC m=+886.587062397" watchObservedRunningTime="2026-11-02 13:59:14.629354465 +0000 UTC m=+886.588730075"
Nov 02 13:59:15.024963 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:15.024199    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:15.024963 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:15.024355    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 13:59:15.024963 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:15.024423    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 13:59:15.299020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:15.298409    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:59:15.299020 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:15.298490    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 13:59:15.350739 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:15.348553    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:15.358968 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:15.358095    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 13:59:15.358968 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:15.358162    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 13:59:15.999125 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:15.999046    6284 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:16.020459 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:16.019992    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 13:59:16.020459 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:16.020069    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 13:59:16.093420 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:16.093065    6284 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5" podUID="f52e51f3-ba95-45c4-a445-98bb17486ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:16.299022 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:16.298413    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:59:16.299022 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:16.298484    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 13:59:16.347054 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:16.347019    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 13:59:16.347346 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:16.347321    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 13:59:17.015915 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:17.015394    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 13:59:17.015915 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:17.015468    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 13:59:17.300474 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:17.300371    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:59:17.301110 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:17.301074    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 13:59:17.344632 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:17.344593    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 13:59:17.344825 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:17.344655    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 13:59:18.013513 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:18.013476    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 13:59:18.013804 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:18.013781    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 13:59:18.301926 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:18.301378    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:59:18.301926 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:18.301443    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 13:59:18.345637 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:18.345580    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 13:59:18.345842 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:18.345671    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 13:59:19.018602 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:19.017805    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 13:59:19.018602 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:19.017867    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 13:59:19.345085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:19.344994    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 13:59:19.345085 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:19.345050    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 13:59:19.486348 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:19.486310    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:19.507972 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:19.488865    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:19.511040 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:19.511006    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 13:59:19.586964 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:19.562995    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" podStartSLOduration=7.56297132 podStartE2EDuration="7.56297132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:14.676812049 +0000 UTC m=+886.636187667" watchObservedRunningTime="2026-11-02 13:59:19.56297132 +0000 UTC m=+891.522346925"
Nov 02 13:59:20.013559 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:20.013518    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 13:59:20.013798 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:20.013577    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 13:59:20.344961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:20.344837    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 13:59:20.344961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:20.344902    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 13:59:20.640709 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:20.640599    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"]
Nov 02 13:59:21.013390 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:21.013351    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 13:59:21.013617 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:21.013418    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 13:59:21.344966 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:21.344830    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 13:59:21.344966 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:21.344905    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 13:59:22.279138 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:22.279088    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:22.280135 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:22.280107    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:22.282397 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:22.282369    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 13:59:22.728453 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:22.728351    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:22.728966 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:22.728783    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:22.729929 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:22.729904    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 13:59:28.090018 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.089975    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"]
Nov 02 13:59:28.094385 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.094356    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.121578 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.121540    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"]
Nov 02 13:59:28.150729 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.150662    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.150902 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.150814    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.151069 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.150997    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.151112 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.151068    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.151112 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.151103    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.151251 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.151228    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.151294 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.151280    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66nd8\" (UniqueName: \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-kube-api-access-66nd8\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.151390 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.151373    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.151443 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.151428    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253094 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253053    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253094 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253101    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253373 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253140    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253373 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253171    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66nd8\" (UniqueName: \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-kube-api-access-66nd8\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253373 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253196    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253373 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253228    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253373 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253273    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253373 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253332    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253685 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253407    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253685 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253564    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253685 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253602    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253831 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253747    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.253898 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.253877    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.254121 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.254102    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.255878 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.255850    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.256048 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.255978    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.278011 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.277972    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66nd8\" (UniqueName: \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-kube-api-access-66nd8\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.279091 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.279064    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.325016 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.324987    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 13:59:28.326498 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.326474    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 13:59:28.412984 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.412857    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:28.734648 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.734612    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"]
Nov 02 13:59:28.737628 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:59:28.737586    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf0c6ad_93b1_4aff_bd98_49aaf2ab01fd.slice/crio-747888e8ebc8e1f386eb85919adb034a627f0420f13e4e9c5041d5127fb0c173 WatchSource:0}: Error finding container 747888e8ebc8e1f386eb85919adb034a627f0420f13e4e9c5041d5127fb0c173: Status 404 returned error can't find the container with id 747888e8ebc8e1f386eb85919adb034a627f0420f13e4e9c5041d5127fb0c173
Nov 02 13:59:28.740602 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.740550    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:28.740741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.740657    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:28.740741 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.740700    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:28.901222 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.901181    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" event={"ID":"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd","Type":"ContainerStarted","Data":"a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731"}
Nov 02 13:59:28.901405 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.901250    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" event={"ID":"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd","Type":"ContainerStarted","Data":"747888e8ebc8e1f386eb85919adb034a627f0420f13e4e9c5041d5127fb0c173"}
Nov 02 13:59:28.945093 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:28.944990    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podStartSLOduration=0.944968242 podStartE2EDuration="944.968242ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:28.94304478 +0000 UTC m=+900.902420387" watchObservedRunningTime="2026-11-02 13:59:28.944968242 +0000 UTC m=+900.904343846"
Nov 02 13:59:29.342973 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.340190    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m" podUID="412c0051-d33b-487c-a56a-fca13299fa75" containerName="main" containerID="cri-o://077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe" gracePeriod=45
Nov 02 13:59:29.413328 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.413289    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:29.414979 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.414928    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 13:59:29.415148 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.415003    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 13:59:29.676782 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.676742    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:59:29.784699 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.784663    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-dshm\") pod \"412c0051-d33b-487c-a56a-fca13299fa75\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") "
Nov 02 13:59:29.784699 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.784711    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/412c0051-d33b-487c-a56a-fca13299fa75-tls-certs\" (UniqueName: \"kubernetes.io/secret/412c0051-d33b-487c-a56a-fca13299fa75-tls-certs\") pod \"412c0051-d33b-487c-a56a-fca13299fa75\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") "
Nov 02 13:59:29.784989 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.784817    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412c0051-d33b-487c-a56a-fca13299fa75-kube-api-access-gr7f9\" (UniqueName: \"kubernetes.io/projected/412c0051-d33b-487c-a56a-fca13299fa75-kube-api-access-gr7f9\") pod \"412c0051-d33b-487c-a56a-fca13299fa75\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") "
Nov 02 13:59:29.784989 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.784851    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-tmp-dir\") pod \"412c0051-d33b-487c-a56a-fca13299fa75\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") "
Nov 02 13:59:29.784989 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.784905    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-home\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-home\") pod \"412c0051-d33b-487c-a56a-fca13299fa75\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") "
Nov 02 13:59:29.784989 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.784963    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-model-cache\") pod \"412c0051-d33b-487c-a56a-fca13299fa75\" (UID: \"412c0051-d33b-487c-a56a-fca13299fa75\") "
Nov 02 13:59:29.785336 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.785297    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-tmp-dir" pod "412c0051-d33b-487c-a56a-fca13299fa75" (UID: "412c0051-d33b-487c-a56a-fca13299fa75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:29.785651 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.785620    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-model-cache" pod "412c0051-d33b-487c-a56a-fca13299fa75" (UID: "412c0051-d33b-487c-a56a-fca13299fa75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:29.785919 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.785883    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-home" pod "412c0051-d33b-487c-a56a-fca13299fa75" (UID: "412c0051-d33b-487c-a56a-fca13299fa75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:29.787196 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.787172    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-dshm" pod "412c0051-d33b-487c-a56a-fca13299fa75" (UID: "412c0051-d33b-487c-a56a-fca13299fa75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:29.787472 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.787454    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412c0051-d33b-487c-a56a-fca13299fa75-tls-certs" pod "412c0051-d33b-487c-a56a-fca13299fa75" (UID: "412c0051-d33b-487c-a56a-fca13299fa75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:29.787604 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.787586    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412c0051-d33b-487c-a56a-fca13299fa75-kube-api-access-gr7f9" pod "412c0051-d33b-487c-a56a-fca13299fa75" (UID: "412c0051-d33b-487c-a56a-fca13299fa75"). InnerVolumeSpecName "kube-api-access-gr7f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:29.887139 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.887046    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:29.887139 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.887081    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/412c0051-d33b-487c-a56a-fca13299fa75-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:29.887139 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.887094    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr7f9\" (UniqueName: \"kubernetes.io/projected/412c0051-d33b-487c-a56a-fca13299fa75-kube-api-access-gr7f9\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:29.887139 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.887103    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:29.887139 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.887111    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:29.887139 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.887119    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/412c0051-d33b-487c-a56a-fca13299fa75-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:29.909065 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.909033    6284 generic.go:356] "Generic (PLEG): container finished" podID="412c0051-d33b-487c-a56a-fca13299fa75" containerID="077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe" exitCode=0
Nov 02 13:59:29.909242 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.909080    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"
Nov 02 13:59:29.909242 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.909119    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m" event={"ID":"412c0051-d33b-487c-a56a-fca13299fa75","Type":"ContainerDied","Data":"077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe"}
Nov 02 13:59:29.909242 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.909162    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m" event={"ID":"412c0051-d33b-487c-a56a-fca13299fa75","Type":"ContainerDied","Data":"0b0432d0030b0543c2931ad05a8b18adc06356d0fc444462bd4115e05c5d2f73"}
Nov 02 13:59:29.909242 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.909191    6284 scope.go:122] "RemoveContainer" containerID="077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe"
Nov 02 13:59:29.919823 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.919798    6284 scope.go:122] "RemoveContainer" containerID="077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe"
Nov 02 13:59:29.920181 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:59:29.920162    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe\": container with ID starting with 077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe not found: ID does not exist" containerID="077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe"
Nov 02 13:59:29.920239 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.920192    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe"} err="failed to get container status \"077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe\": rpc error: code = NotFound desc = could not find container \"077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe\": container with ID starting with 077fc6f320d8788f7d8115218bdb8f9177d03ab7225cfd5658f0ebdeb3650cbe not found: ID does not exist"
Nov 02 13:59:29.978699 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.978647    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"]
Nov 02 13:59:29.982428 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:29.982392    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-b-409544bc-kserve-8576d6b68b-cdv9m"]
Nov 02 13:59:30.183508 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:30.183408    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412c0051-d33b-487c-a56a-fca13299fa75" path="/var/lib/kubelet/pods/412c0051-d33b-487c-a56a-fca13299fa75/volumes"
Nov 02 13:59:30.413718 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:30.413678    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 13:59:30.414167 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:30.413754    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 13:59:31.413436 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:31.413366    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 13:59:31.413436 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:31.413436    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 13:59:32.416961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:32.416155    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 13:59:32.416961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:32.416223    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 13:59:33.416462 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:33.416411    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 13:59:33.416792 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:33.416759    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 13:59:34.414276 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:34.414238    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 13:59:34.414759 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:34.414297    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 13:59:35.414271 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:35.414226    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 13:59:35.414475 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:35.414286    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 13:59:35.706456 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:35.706359    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5" podUID="f52e51f3-ba95-45c4-a445-98bb17486ed7" containerName="main" containerID="cri-o://a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904" gracePeriod=45
Nov 02 13:59:35.710471 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:35.710438    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5" podUID="f52e51f3-ba95-45c4-a445-98bb17486ed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": read tcp 10.134.0.2:51830->10.134.0.54:8000: read: connection reset by peer"
Nov 02 13:59:36.035174 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.035140    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:59:36.147872 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.147833    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-home\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-home\") pod \"f52e51f3-ba95-45c4-a445-98bb17486ed7\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") "
Nov 02 13:59:36.148057 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.147898    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52e51f3-ba95-45c4-a445-98bb17486ed7-kube-api-access-458h4\" (UniqueName: \"kubernetes.io/projected/f52e51f3-ba95-45c4-a445-98bb17486ed7-kube-api-access-458h4\") pod \"f52e51f3-ba95-45c4-a445-98bb17486ed7\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") "
Nov 02 13:59:36.148057 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.148013    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-tmp-dir\") pod \"f52e51f3-ba95-45c4-a445-98bb17486ed7\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") "
Nov 02 13:59:36.148057 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.148054    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-model-cache\") pod \"f52e51f3-ba95-45c4-a445-98bb17486ed7\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") "
Nov 02 13:59:36.148179 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.148099    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f52e51f3-ba95-45c4-a445-98bb17486ed7-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52e51f3-ba95-45c4-a445-98bb17486ed7-tls-certs\") pod \"f52e51f3-ba95-45c4-a445-98bb17486ed7\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") "
Nov 02 13:59:36.148179 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.148131    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-dshm\") pod \"f52e51f3-ba95-45c4-a445-98bb17486ed7\" (UID: \"f52e51f3-ba95-45c4-a445-98bb17486ed7\") "
Nov 02 13:59:36.148245 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.148169    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-home" pod "f52e51f3-ba95-45c4-a445-98bb17486ed7" (UID: "f52e51f3-ba95-45c4-a445-98bb17486ed7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:36.148324 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.148302    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-tmp-dir" pod "f52e51f3-ba95-45c4-a445-98bb17486ed7" (UID: "f52e51f3-ba95-45c4-a445-98bb17486ed7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:36.148389 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.148317    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-model-cache" pod "f52e51f3-ba95-45c4-a445-98bb17486ed7" (UID: "f52e51f3-ba95-45c4-a445-98bb17486ed7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:36.148582 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.148562    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:36.148677 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.148601    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:36.148677 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.148612    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:36.150372 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.150335    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f52e51f3-ba95-45c4-a445-98bb17486ed7-tls-certs" pod "f52e51f3-ba95-45c4-a445-98bb17486ed7" (UID: "f52e51f3-ba95-45c4-a445-98bb17486ed7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:36.150491 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.150382    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-dshm" pod "f52e51f3-ba95-45c4-a445-98bb17486ed7" (UID: "f52e51f3-ba95-45c4-a445-98bb17486ed7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:59:36.150608 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.150589    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52e51f3-ba95-45c4-a445-98bb17486ed7-kube-api-access-458h4" pod "f52e51f3-ba95-45c4-a445-98bb17486ed7" (UID: "f52e51f3-ba95-45c4-a445-98bb17486ed7"). InnerVolumeSpecName "kube-api-access-458h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:36.249725 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.249679    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f52e51f3-ba95-45c4-a445-98bb17486ed7-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:36.249725 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.249728    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f52e51f3-ba95-45c4-a445-98bb17486ed7-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:36.250087 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.249745    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458h4\" (UniqueName: \"kubernetes.io/projected/f52e51f3-ba95-45c4-a445-98bb17486ed7-kube-api-access-458h4\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 13:59:36.253253 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.253223    6284 generic.go:356] "Generic (PLEG): container finished" podID="f52e51f3-ba95-45c4-a445-98bb17486ed7" containerID="a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904" exitCode=0
Nov 02 13:59:36.253425 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.253290    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"
Nov 02 13:59:36.253425 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.253318    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5" event={"ID":"f52e51f3-ba95-45c4-a445-98bb17486ed7","Type":"ContainerDied","Data":"a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904"}
Nov 02 13:59:36.253425 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.253367    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5" event={"ID":"f52e51f3-ba95-45c4-a445-98bb17486ed7","Type":"ContainerDied","Data":"5a48999d5e3916d0f816b931eb18769e4feb194de72f74faa038e5e9a18e84eb"}
Nov 02 13:59:36.253425 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.253388    6284 scope.go:122] "RemoveContainer" containerID="a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904"
Nov 02 13:59:36.264726 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.264699    6284 scope.go:122] "RemoveContainer" containerID="a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904"
Nov 02 13:59:36.265200 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:59:36.265171    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904\": container with ID starting with a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904 not found: ID does not exist" containerID="a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904"
Nov 02 13:59:36.265319 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.265208    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904"} err="failed to get container status \"a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904\": rpc error: code = NotFound desc = could not find container \"a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904\": container with ID starting with a885f24ad38dee3ddb4c3212357bbdfb1a53af6e0a24d2e593f266ec52663904 not found: ID does not exist"
Nov 02 13:59:36.304470 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.304422    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"]
Nov 02 13:59:36.310345 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.310305    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/e2e-conflict-a-409544bc-kserve-6cc545ccb8-jcdx5"]
Nov 02 13:59:36.413541 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.413505    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 13:59:36.413735 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:36.413565    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 13:59:37.526296 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:37.526263    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:37.526897 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:37.526873    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:37.528237 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:37.528210    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 13:59:38.172248 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:38.172205    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52e51f3-ba95-45c4-a445-98bb17486ed7" path="/var/lib/kubelet/pods/f52e51f3-ba95-45c4-a445-98bb17486ed7/volumes"
Nov 02 13:59:40.153858 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.153822    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"]
Nov 02 13:59:40.154468 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.154450    6284 cpu_manager.go:414] "containerMap: removing container" podUID="412c0051-d33b-487c-a56a-fca13299fa75" containerName="main"
Nov 02 13:59:40.154514 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.154468    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412c0051-d33b-487c-a56a-fca13299fa75" containerName="main"
Nov 02 13:59:40.154546 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.154518    6284 cpu_manager.go:414] "containerMap: removing container" podUID="f52e51f3-ba95-45c4-a445-98bb17486ed7" containerName="main"
Nov 02 13:59:40.154546 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.154524    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52e51f3-ba95-45c4-a445-98bb17486ed7" containerName="main"
Nov 02 13:59:40.154633 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.154620    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52e51f3-ba95-45c4-a445-98bb17486ed7" containerName="main"
Nov 02 13:59:40.154679 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.154640    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="412c0051-d33b-487c-a56a-fca13299fa75" containerName="main"
Nov 02 13:59:40.159727 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.159695    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.190672 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.190640    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"openshift-service-ca.crt\""
Nov 02 13:59:40.191533 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.190714    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"default-dockercfg-clf8h\""
Nov 02 13:59:40.191533 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.190640    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"kube-root-ca.crt\""
Nov 02 13:59:40.191533 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.190890    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"distinct-w0-1a562c-kserve-self-signed-certs\""
Nov 02 13:59:40.194466 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.194435    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"]
Nov 02 13:59:40.209121 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.209079    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-home\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.209334 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.209145    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-dshm\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.209542 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.209508    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca46c70-2ef9-4179-be04-6469e731b1d8-tls-certs\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.209623 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.209605    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-model-cache\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.209706 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.209689    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwpt\" (UniqueName: \"kubernetes.io/projected/fca46c70-2ef9-4179-be04-6469e731b1d8-kube-api-access-tqwpt\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.209811 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.209792    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-tmp-dir\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.310560 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.310501    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-dshm\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.310560 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.310563    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca46c70-2ef9-4179-be04-6469e731b1d8-tls-certs\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.310848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.310593    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-model-cache\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.310848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.310654    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwpt\" (UniqueName: \"kubernetes.io/projected/fca46c70-2ef9-4179-be04-6469e731b1d8-kube-api-access-tqwpt\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.310848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.310697    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-tmp-dir\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.310848 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.310843    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-home\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.311183 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.311153    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-model-cache\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.311261 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.311207    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-home\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.311411 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.311392    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-tmp-dir\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.313623 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.313588    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-dshm\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.313991 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.313972    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca46c70-2ef9-4179-be04-6469e731b1d8-tls-certs\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.332503 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.332470    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwpt\" (UniqueName: \"kubernetes.io/projected/fca46c70-2ef9-4179-be04-6469e731b1d8-kube-api-access-tqwpt\") pod \"distinct-w0-1a562c-kserve-54cc7ccd67-9rn95\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.482293 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.482201    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:40.809334 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:40.809305    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"]
Nov 02 13:59:40.812890 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:59:40.812841    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfca46c70_2ef9_4179_be04_6469e731b1d8.slice/crio-05c465d895c14a5133b74f249efcd9e987e5538b1c6dcfd5a5bf07e567dc738c WatchSource:0}: Error finding container 05c465d895c14a5133b74f249efcd9e987e5538b1c6dcfd5a5bf07e567dc738c: Status 404 returned error can't find the container with id 05c465d895c14a5133b74f249efcd9e987e5538b1c6dcfd5a5bf07e567dc738c
Nov 02 13:59:41.300749 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:41.300707    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" event={"ID":"fca46c70-2ef9-4179-be04-6469e731b1d8","Type":"ContainerStarted","Data":"3ca49d79caefd8eacf94da8dc2c850cac685588ba1276ac1e3e029e21f5a5665"}
Nov 02 13:59:41.300749 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:41.300750    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" event={"ID":"fca46c70-2ef9-4179-be04-6469e731b1d8","Type":"ContainerStarted","Data":"05c465d895c14a5133b74f249efcd9e987e5538b1c6dcfd5a5bf07e567dc738c"}
Nov 02 13:59:41.338135 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:41.337640    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podStartSLOduration=1.337617444 podStartE2EDuration="1.337617444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:41.336763057 +0000 UTC m=+913.296138665" watchObservedRunningTime="2026-11-02 13:59:41.337617444 +0000 UTC m=+913.296993052"
Nov 02 13:59:41.482546 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:41.482508    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:43.789738 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:43.789694    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"]
Nov 02 13:59:43.806481 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:43.806450    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:43.828965 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:43.826901    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ad42e0-openshift-default-dockercfg-gmvz9\""
Nov 02 13:59:43.828965 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:43.827981    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"]
Nov 02 13:59:44.040017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.036483    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.040017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.036529    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.040017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.036589    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.040017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.036623    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqllb\" (UniqueName: \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-kube-api-access-hqllb\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.040017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.036673    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.040017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.036707    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cc467e-61a3-4f4b-b317-60fe7deffb63-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.040017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.036729    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.040017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.036882    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.040017 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.036911    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.156969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.156572    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.156969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.156625    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.156969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.156687    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.156969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.156713    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.156969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.156763    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.156969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.156797    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqllb\" (UniqueName: \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-kube-api-access-hqllb\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.156969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.156849    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.156969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.156895    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cc467e-61a3-4f4b-b317-60fe7deffb63-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.156969 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.156954    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.157571 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.157525    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.158213 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.157784    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.173961 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.159169    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.175184 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.160444    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.175395 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.161115    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cc467e-61a3-4f4b-b317-60fe7deffb63-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.287962 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.238431    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.287962 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.168926    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.293023 ip-10-0-131-98 kubenswrapper[6284]: E1102 13:59:44.288404    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 13:59:44.314512 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.314476    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.317760 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.317721    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqllb\" (UniqueName: \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-kube-api-access-hqllb\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.461319 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.461270    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:44.868886 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.868790    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"]
Nov 02 13:59:44.872112 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:59:44.872079    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cc467e_61a3_4f4b_b317_60fe7deffb63.slice/crio-46137e497ae7c317fff790f96ea93578ded0d7bb69098035f50d38b85f57ef7f WatchSource:0}: Error finding container 46137e497ae7c317fff790f96ea93578ded0d7bb69098035f50d38b85f57ef7f: Status 404 returned error can't find the container with id 46137e497ae7c317fff790f96ea93578ded0d7bb69098035f50d38b85f57ef7f
Nov 02 13:59:44.874839 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.874799    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:44.874983 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.874873    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:44.874983 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:44.874905    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:45.367223 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:45.367174    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" event={"ID":"94cc467e-61a3-4f4b-b317-60fe7deffb63","Type":"ContainerStarted","Data":"bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a"}
Nov 02 13:59:45.367428 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:45.367254    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" event={"ID":"94cc467e-61a3-4f4b-b317-60fe7deffb63","Type":"ContainerStarted","Data":"46137e497ae7c317fff790f96ea93578ded0d7bb69098035f50d38b85f57ef7f"}
Nov 02 13:59:45.418790 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:45.418727    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" podStartSLOduration=2.41870686 podStartE2EDuration="2.41870686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:45.406555666 +0000 UTC m=+917.365931286" watchObservedRunningTime="2026-11-02 13:59:45.41870686 +0000 UTC m=+917.378082467"
Nov 02 13:59:45.462461 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:45.462420    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:45.464246 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:45.464217    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 13:59:45.464346 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:45.464299    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 13:59:46.466426 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:46.466383    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 13:59:46.467225 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:46.467190    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 13:59:47.472138 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:47.472098    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 13:59:47.473239 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:47.472769    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 13:59:48.463423 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:48.463381    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 13:59:48.463745 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:48.463706    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 13:59:49.468454 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:49.468416    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 13:59:49.469107 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:49.469071    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 13:59:50.472502 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:50.468702    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 13:59:50.472502 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:50.468773    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 13:59:50.483403 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:50.483357    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:50.497358 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:50.497328    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:51.427851 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:51.427812    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 13:59:51.462609 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:51.462573    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 13:59:51.462802 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:51.462641    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 13:59:52.467571 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:52.467534    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:52.468114 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:52.467957    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:52.468747 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:52.468726    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 13:59:58.187290 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.187251    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"]
Nov 02 13:59:58.191452 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.191421    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.220133 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.220090    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"]
Nov 02 13:59:58.282987 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.282924    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.283202 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.283060    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.283202 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.283124    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.283202 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.283166    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.283202 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.283195    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.283403 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.283326    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cbl9\" (UniqueName: \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-kube-api-access-8cbl9\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.283403 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.283385    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.283522 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.283506    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.283586 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.283542    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844a58bb-ce76-46ff-ab23-d86b373c61fb-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.385151 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385112    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844a58bb-ce76-46ff-ab23-d86b373c61fb-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.385350 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385274    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.385350 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385334    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.385549 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385518    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.385782 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385756    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.385931 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385796    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.385931 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385833    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.385931 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385857    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbl9\" (UniqueName: \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-kube-api-access-8cbl9\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.385931 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385873    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.385931 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385901    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.386225 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385951    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844a58bb-ce76-46ff-ab23-d86b373c61fb-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.386225 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.385981    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.386225 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.386121    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.386418 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.386387    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.388463 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.388436    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.388723 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.388703    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.406595 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.406565    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.406757 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.406631    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbl9\" (UniqueName: \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-kube-api-access-8cbl9\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.515506 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.515466    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:58.846257 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.846114    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"]
Nov 02 13:59:58.850540 ip-10-0-131-98 kubenswrapper[6284]: W1102 13:59:58.850506    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844a58bb_ce76_46ff_ab23_d86b373c61fb.slice/crio-51b11fbb82c23f7debd9852cedcb265b97b61f96132cb1fe6fcf020a9dfda41b WatchSource:0}: Error finding container 51b11fbb82c23f7debd9852cedcb265b97b61f96132cb1fe6fcf020a9dfda41b: Status 404 returned error can't find the container with id 51b11fbb82c23f7debd9852cedcb265b97b61f96132cb1fe6fcf020a9dfda41b
Nov 02 13:59:58.853285 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.853251    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:58.853389 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.853321    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:58.853389 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:58.853350    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 13:59:59.483633 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:59.483592    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" event={"ID":"844a58bb-ce76-46ff-ab23-d86b373c61fb","Type":"ContainerStarted","Data":"6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06"}
Nov 02 13:59:59.484135 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:59.483659    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" event={"ID":"844a58bb-ce76-46ff-ab23-d86b373c61fb","Type":"ContainerStarted","Data":"51b11fbb82c23f7debd9852cedcb265b97b61f96132cb1fe6fcf020a9dfda41b"}
Nov 02 13:59:59.516958 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:59.516118    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 13:59:59.523581 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:59.523539    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 13:59:59.523798 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:59.523635    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 13:59:59.585065 ip-10-0-131-98 kubenswrapper[6284]: I1102 13:59:59.584989    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podStartSLOduration=1.5849645890000001 podStartE2EDuration="1.584964589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:59.568072989 +0000 UTC m=+931.527448595" watchObservedRunningTime="2026-11-02 13:59:59.584964589 +0000 UTC m=+931.544340199"
Nov 02 14:00:00.519868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:00.519826    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 14:00:00.520530 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:00.520501    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 14:00:01.553513 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:01.553473    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 14:00:01.554234 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:01.554200    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 14:00:02.602959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:02.577350    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 14:00:02.602959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:02.577416    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 14:00:03.591962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:03.581447    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 14:00:03.591962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:03.581517    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 14:00:03.650047 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:03.650000    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"]
Nov 02 14:00:03.884559 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:03.884479    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"]
Nov 02 14:00:03.979974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:03.884841    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:03.979974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:03.967736    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efe3c946-l2gzk\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:03.979974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:03.972438    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efe3c946-l2gzk\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:03.979974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:03.972489    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjsd\" (UniqueName: \"kubernetes.io/projected/1b31a177-eabd-42ff-8ffb-9044c2f936f9-kube-api-access-4zjsd\") pod \"maas-api-revoke-sub-efe3c946-l2gzk\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:04.063363 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.063333    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:00:04.135530 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.067275    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:04.135785 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.110087    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efe3c946-l2gzk\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:04.136008 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.135985    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjsd\" (UniqueName: \"kubernetes.io/projected/1b31a177-eabd-42ff-8ffb-9044c2f936f9-kube-api-access-4zjsd\") pod \"maas-api-revoke-sub-efe3c946-l2gzk\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:04.136342 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.136323    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efe3c946-l2gzk\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:04.137362 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.137338    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efe3c946-l2gzk\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:04.186546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.186508    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:04.186964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.186788    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.287162 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.287122    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efe3c946-l2gzk\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:04.359963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.326743    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjsd\" (UniqueName: \"kubernetes.io/projected/1b31a177-eabd-42ff-8ffb-9044c2f936f9-kube-api-access-4zjsd\") pod \"maas-api-revoke-sub-efe3c946-l2gzk\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:04.454013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.453458    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:04.588329 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.588252    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.616199 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.614361    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 14:00:04.616199 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.614423    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 14:00:04.755003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.754958    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"]
Nov 02 14:00:05.008033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:04.900046    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 14:00:05.272670 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.272574    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"]
Nov 02 14:00:05.286484 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.286446    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"]
Nov 02 14:00:05.287125 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.287098    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.290170 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.290126    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.309789 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.309727    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"]
Nov 02 14:00:05.415834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.406480    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.415834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.406553    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.415834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.406602    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s59rr\" (UniqueName: \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-kube-api-access-s59rr\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.415834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.406638    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.415834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.406726    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.415834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.406828    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.415834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.406884    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.415834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.406999    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/366922ef-5653-4b0f-9f15-4cd7f74d7528-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.415834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.407107    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.525714    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/366922ef-5653-4b0f-9f15-4cd7f74d7528-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.525772    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.525799    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmpkv\" (UniqueName: \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-kube-api-access-fmpkv\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.525895    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.525949    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.525989    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526014    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526041    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526080    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526113    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s59rr\" (UniqueName: \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-kube-api-access-s59rr\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526138    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526168    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526265    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526298    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526325    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526393    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526487    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.526541    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.527127    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.539735    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/366922ef-5653-4b0f-9f15-4cd7f74d7528-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.540155    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.540430    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.540691    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.554314    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 14:00:05.555493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.554400    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 14:00:05.611006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.593280    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:05.611006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.593745    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.811596    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.834617    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.834667    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpkv\" (UniqueName: \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-kube-api-access-fmpkv\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.834763    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.834830    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.834872    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.834904    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.835024    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.835054    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.835118    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.836014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.835733    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.837058 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.836147    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s59rr\" (UniqueName: \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-kube-api-access-s59rr\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.879116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.838400    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.879116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.839169    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.879116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.839299    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.879116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.840499    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.934124 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.891475    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.934124 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.930748    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpkv\" (UniqueName: \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-kube-api-access-fmpkv\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.935559 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.935459    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:00:05.937129 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.937069    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.937674 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.937632    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:05.944341 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.942318    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:00:05.944341 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.943311    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.957387 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.957313    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:05.989985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:05.988208    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"]
Nov 02 14:00:06.089709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.089613    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk" event={"ID":"1b31a177-eabd-42ff-8ffb-9044c2f936f9","Type":"ContainerStarted","Data":"d850d056a2d13d8c2f93b7eb1d3e55348240cc0b3a7d2768c871ffc376ee5023"}
Nov 02 14:00:06.110993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.110916    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:06.244948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.244897    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:06.479511 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:00:06.479474    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366922ef_5653_4b0f_9f15_4cd7f74d7528.slice/crio-ab7b0e11d91a76756cf12c48b348f417d7d39a522a9b331a08d35509f7bfb624 WatchSource:0}: Error finding container ab7b0e11d91a76756cf12c48b348f417d7d39a522a9b331a08d35509f7bfb624: Status 404 returned error can't find the container with id ab7b0e11d91a76756cf12c48b348f417d7d39a522a9b331a08d35509f7bfb624
Nov 02 14:00:06.480428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.480403    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"]
Nov 02 14:00:06.482200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.482168    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:00:06.482255 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.482236    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:00:06.482290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.482267    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:00:06.517904 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.517862    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 14:00:06.518105 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.517928    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 14:00:06.589671 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.589639    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"]
Nov 02 14:00:06.591322 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:00:06.591289    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc244ec7c_9329_4f67_bb8d_cb67da5ec034.slice/crio-f0cfcd025a45aa561d57f916fbad901653fc4f9389e30586001eac195002779c WatchSource:0}: Error finding container f0cfcd025a45aa561d57f916fbad901653fc4f9389e30586001eac195002779c: Status 404 returned error can't find the container with id f0cfcd025a45aa561d57f916fbad901653fc4f9389e30586001eac195002779c
Nov 02 14:00:06.594461 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.594418    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:00:06.594572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.594518    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:00:06.594572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:06.594562    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:00:07.099630 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.099587    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" event={"ID":"c244ec7c-9329-4f67-bb8d-cb67da5ec034","Type":"ContainerStarted","Data":"c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228"}
Nov 02 14:00:07.100177 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.099658    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" event={"ID":"c244ec7c-9329-4f67-bb8d-cb67da5ec034","Type":"ContainerStarted","Data":"f0cfcd025a45aa561d57f916fbad901653fc4f9389e30586001eac195002779c"}
Nov 02 14:00:07.101571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.101540    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" event={"ID":"366922ef-5653-4b0f-9f15-4cd7f74d7528","Type":"ContainerStarted","Data":"3d31d56bd471e552c6d8e8bf75ebcc9a71d44f5eec764c7b6b97333977a6962e"}
Nov 02 14:00:07.101751 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.101590    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" event={"ID":"366922ef-5653-4b0f-9f15-4cd7f74d7528","Type":"ContainerStarted","Data":"ab7b0e11d91a76756cf12c48b348f417d7d39a522a9b331a08d35509f7bfb624"}
Nov 02 14:00:07.111110 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.111070    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:07.113250 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.113219    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:00:07.113398 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.113282    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:00:07.143697 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.143637    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podStartSLOduration=3.143620554 podStartE2EDuration="3.143620554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:07.140238409 +0000 UTC m=+939.099614016" watchObservedRunningTime="2026-11-02 14:00:07.143620554 +0000 UTC m=+939.102996157"
Nov 02 14:00:07.188426 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.188354    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" podStartSLOduration=3.188333234 podStartE2EDuration="3.188333234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:07.185647224 +0000 UTC m=+939.145022832" watchObservedRunningTime="2026-11-02 14:00:07.188333234 +0000 UTC m=+939.147708843"
Nov 02 14:00:07.245325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.245284    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:07.247059 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.247024    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:00:07.247256 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.247092    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:00:07.525460 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.524394    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 14:00:07.525460 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:07.524467    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 14:00:08.111326 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:08.111282    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:00:08.111848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:08.111334    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:00:08.245475 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:08.245427    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:00:08.245684 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:08.245508    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:00:08.515918 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:08.515876    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 14:00:08.516266 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:08.516235    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 14:00:09.115964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:09.115176    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:00:09.115964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:09.115244    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:00:09.246466 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:09.246425    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:00:09.246698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:09.246495    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:00:10.112961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:10.112599    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:00:10.112961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:10.112661    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:00:10.245528 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:10.245495    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:00:10.245959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:10.245545    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:00:10.521906 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:10.521858    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:10.522153 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:10.521928    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:11.111397 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:11.111355    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:00:11.111710 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:11.111675    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:00:11.246307 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:11.246266    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:00:11.246748 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:11.246327    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:00:11.564962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:11.564129    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:11.564962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:11.564203    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:11.639328 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:11.639289    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 14:00:11.640289 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:11.640267    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 14:00:11.647492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:11.647465    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 14:00:12.111810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:12.111756    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:00:12.112071 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:12.111837    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:00:12.246233 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:12.246187    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:00:12.246447 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:12.246261    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:00:13.112702 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:13.112311    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:00:13.112702 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:13.112384    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:00:13.246399 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:13.246363    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:00:13.246572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:13.246436    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:00:13.899535 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:00:13.899491    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:00:14.112516 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:14.112474    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 14:00:14.112766 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:14.112542    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 14:00:14.246106 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:14.246068    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 14:00:14.246310 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:14.246152    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 14:00:15.115979 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:15.115929    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:15.116393 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:15.116257    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:15.117078 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:15.117055    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:00:15.193296 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:15.193199    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk" event={"ID":"1b31a177-eabd-42ff-8ffb-9044c2f936f9","Type":"ContainerStarted","Data":"e8bc71ff37577a65deefa0d0f7cb83f01b7aa33a31bd0853658452a9bf413ed4"}
Nov 02 14:00:15.247700 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:15.247631    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk" podStartSLOduration=3.675397393 podStartE2EDuration="12.247611476s" totalImagesPullingTime="8.572214083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:15.246156486 +0000 UTC m=+947.205532092" watchObservedRunningTime="2026-11-02 14:00:15.247611476 +0000 UTC m=+947.206987084"
Nov 02 14:00:15.254459 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:15.254427    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:15.254959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:15.254912    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:15.256117 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:15.256092    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:00:16.201662 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:16.201627    6284 generic.go:356] "Generic (PLEG): container finished" podID="1b31a177-eabd-42ff-8ffb-9044c2f936f9" containerID="e8bc71ff37577a65deefa0d0f7cb83f01b7aa33a31bd0853658452a9bf413ed4" exitCode=0
Nov 02 14:00:16.202125 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:16.201706    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk" event={"ID":"1b31a177-eabd-42ff-8ffb-9044c2f936f9","Type":"ContainerDied","Data":"e8bc71ff37577a65deefa0d0f7cb83f01b7aa33a31bd0853658452a9bf413ed4"}
Nov 02 14:00:17.369890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:17.369849    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:17.508113 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:17.508067    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-xks-ca\") pod \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") "
Nov 02 14:00:17.508339 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:17.508298    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-openshift-ca\") pod \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") "
Nov 02 14:00:17.508339 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:17.508326    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b31a177-eabd-42ff-8ffb-9044c2f936f9-kube-api-access-4zjsd\" (UniqueName: \"kubernetes.io/projected/1b31a177-eabd-42ff-8ffb-9044c2f936f9-kube-api-access-4zjsd\") pod \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\" (UID: \"1b31a177-eabd-42ff-8ffb-9044c2f936f9\") "
Nov 02 14:00:17.508666 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:17.508641    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-openshift-ca" pod "1b31a177-eabd-42ff-8ffb-9044c2f936f9" (UID: "1b31a177-eabd-42ff-8ffb-9044c2f936f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:17.508790 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:17.508772    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-openshift-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:00:17.510516 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:17.510481    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-xks-ca" pod "1b31a177-eabd-42ff-8ffb-9044c2f936f9" (UID: "1b31a177-eabd-42ff-8ffb-9044c2f936f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:17.510790 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:17.510765    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b31a177-eabd-42ff-8ffb-9044c2f936f9-kube-api-access-4zjsd" pod "1b31a177-eabd-42ff-8ffb-9044c2f936f9" (UID: "1b31a177-eabd-42ff-8ffb-9044c2f936f9"). InnerVolumeSpecName "kube-api-access-4zjsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:17.609895 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:17.609842    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjsd\" (UniqueName: \"kubernetes.io/projected/1b31a177-eabd-42ff-8ffb-9044c2f936f9-kube-api-access-4zjsd\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:00:17.609895 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:17.609880    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b31a177-eabd-42ff-8ffb-9044c2f936f9-maas-api-xks-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:00:18.216920 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:18.216886    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"
Nov 02 14:00:18.216920 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:18.216899    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk" event={"ID":"1b31a177-eabd-42ff-8ffb-9044c2f936f9","Type":"ContainerDied","Data":"d850d056a2d13d8c2f93b7eb1d3e55348240cc0b3a7d2768c871ffc376ee5023"}
Nov 02 14:00:18.217175 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:18.216965    6284 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d850d056a2d13d8c2f93b7eb1d3e55348240cc0b3a7d2768c871ffc376ee5023"
Nov 02 14:00:22.670339 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.670297    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"]
Nov 02 14:00:22.671116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.671094    6284 cpu_manager.go:414] "containerMap: removing container" podUID="1b31a177-eabd-42ff-8ffb-9044c2f936f9" containerName="revoke-keys"
Nov 02 14:00:22.671232 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.671118    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b31a177-eabd-42ff-8ffb-9044c2f936f9" containerName="revoke-keys"
Nov 02 14:00:22.671295 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.671260    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b31a177-eabd-42ff-8ffb-9044c2f936f9" containerName="revoke-keys"
Nov 02 14:00:22.675625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.675593    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.692352 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.692314    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"]
Nov 02 14:00:22.697444 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.697415    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"distinct-2-w0-1a562c-kserve-self-signed-certs\""
Nov 02 14:00:22.783204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.783168    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-model-cache\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.783399 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.783314    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062cbb9f-9666-434b-bff9-d1e976cdd550-tls-certs\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.783475 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.783396    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-home\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.783562 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.783538    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-tmp-dir\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.783626 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.783576    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwphz\" (UniqueName: \"kubernetes.io/projected/062cbb9f-9666-434b-bff9-d1e976cdd550-kube-api-access-gwphz\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.783730 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.783717    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-dshm\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.886020 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.885171    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-dshm\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.886020 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.885272    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-model-cache\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.886020 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.885397    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062cbb9f-9666-434b-bff9-d1e976cdd550-tls-certs\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.886020 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.885466    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-home\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.886020 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.885574    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-tmp-dir\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.886020 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.885598    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwphz\" (UniqueName: \"kubernetes.io/projected/062cbb9f-9666-434b-bff9-d1e976cdd550-kube-api-access-gwphz\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.887437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.886907    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-tmp-dir\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.888148 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.888089    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-model-cache\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.888502 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.888479    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-home\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.895236 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.895156    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-dshm\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.901641 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.901609    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062cbb9f-9666-434b-bff9-d1e976cdd550-tls-certs\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:22.922753 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:22.922672    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwphz\" (UniqueName: \"kubernetes.io/projected/062cbb9f-9666-434b-bff9-d1e976cdd550-kube-api-access-gwphz\") pod \"distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") " pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:23.001243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:23.001198    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:23.323103 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:23.323070    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"]
Nov 02 14:00:23.325580 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:00:23.325545    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod062cbb9f_9666_434b_bff9_d1e976cdd550.slice/crio-609c4651960aef5a61abbe94cca144ab03621c3235048566d2136cf0358e2383 WatchSource:0}: Error finding container 609c4651960aef5a61abbe94cca144ab03621c3235048566d2136cf0358e2383: Status 404 returned error can't find the container with id 609c4651960aef5a61abbe94cca144ab03621c3235048566d2136cf0358e2383
Nov 02 14:00:24.265930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:24.265881    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" event={"ID":"062cbb9f-9666-434b-bff9-d1e976cdd550","Type":"ContainerStarted","Data":"2216cf8a8c3a18028461f3053086f5d512566825870a0dadca04fdff8d401069"}
Nov 02 14:00:24.266492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:24.265985    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" event={"ID":"062cbb9f-9666-434b-bff9-d1e976cdd550","Type":"ContainerStarted","Data":"609c4651960aef5a61abbe94cca144ab03621c3235048566d2136cf0358e2383"}
Nov 02 14:00:24.306998 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:24.306917    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" podStartSLOduration=2.30689702 podStartE2EDuration="2.30689702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:24.302806255 +0000 UTC m=+956.262181865" watchObservedRunningTime="2026-11-02 14:00:24.30689702 +0000 UTC m=+956.266272626"
Nov 02 14:00:25.007370 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:25.007339    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:26.004966 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:25.997062    6284 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:27.758962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:27.748108    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:27.758962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:27.748254    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:28.124300 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:28.124185    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:28.126097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:28.126056    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:28.672018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:28.668131    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hwr8w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:28.672018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:28.668222    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:29.019566 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:29.019521    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:29.020424 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:29.020375    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:29.162016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:29.137412    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 14:00:29.162016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:29.161930    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 14:00:29.389960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:29.349177    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 02 14:00:29.389960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:29.349836    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:00:29.748963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:29.743521    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 14:00:30.305972 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:30.305920    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:00:33.004636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:33.004595    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:33.017906 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:33.017868    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:33.736140 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:33.736107    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:43.897663 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:00:43.897624    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:00:44.125337 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.125297    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"]
Nov 02 14:00:44.134664 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.134626    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.175526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.175439    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"]
Nov 02 14:00:44.234717 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.234684    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv27v\" (UniqueName: \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-kube-api-access-nv27v\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.234901 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.234743    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.234901 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.234863    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.235090 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.235014    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.235090 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.235033    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.235299 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.235271    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.235529 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.235502    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.235709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.235686    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.235753 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.235734    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.337273 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337224    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.337495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337318    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.337495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337339    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.337495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337400    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.337495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337420    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.337728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337525    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.337728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337560    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.337728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337600    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv27v\" (UniqueName: \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-kube-api-access-nv27v\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.337728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337663    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.338009 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337778    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.338009 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337819    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.338009 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.337903    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.338260 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.338236    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.338354 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.338339    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.340067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.340044    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.340537 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.340510    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.360886 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.360848    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.362599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.362560    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv27v\" (UniqueName: \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-kube-api-access-nv27v\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.454382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.454275    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:44.785091 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.785057    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"]
Nov 02 14:00:44.788173 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:00:44.788115    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c9ecb9_4cd3_432d_8f15_a7af2719a972.slice/crio-36c7500e9413bb279a0045d65bab44133b009ea4fdb747e7ebec2e7e17af982d WatchSource:0}: Error finding container 36c7500e9413bb279a0045d65bab44133b009ea4fdb747e7ebec2e7e17af982d: Status 404 returned error can't find the container with id 36c7500e9413bb279a0045d65bab44133b009ea4fdb747e7ebec2e7e17af982d
Nov 02 14:00:44.791367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.791321    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:00:44.791576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.791429    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:00:44.791576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.791498    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:00:44.804682 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:44.804643    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" event={"ID":"02c9ecb9-4cd3-432d-8f15-a7af2719a972","Type":"ContainerStarted","Data":"36c7500e9413bb279a0045d65bab44133b009ea4fdb747e7ebec2e7e17af982d"}
Nov 02 14:00:45.813582 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:45.813545    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" event={"ID":"02c9ecb9-4cd3-432d-8f15-a7af2719a972","Type":"ContainerStarted","Data":"f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da"}
Nov 02 14:00:45.883988 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:45.883901    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podStartSLOduration=1.883876898 podStartE2EDuration="1.883876898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:45.870491699 +0000 UTC m=+977.829867308" watchObservedRunningTime="2026-11-02 14:00:45.883876898 +0000 UTC m=+977.843252506"
Nov 02 14:00:46.299053 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.298999    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"]
Nov 02 14:00:46.306498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.306464    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.320850 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.320815    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d776122a-serving-cert\""
Nov 02 14:00:46.326045 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.326002    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"]
Nov 02 14:00:46.381067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.381032    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-maas-api-tls\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.381281 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.381242    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-metrics-certs\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.381541 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.381519    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq7mn\" (UniqueName: \"kubernetes.io/projected/095b057e-1829-43a4-bcac-4ad5c5e6851d-kube-api-access-hq7mn\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.381697 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.381680    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/095b057e-1829-43a4-bcac-4ad5c5e6851d-openshift-service-ca\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.472380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.472334    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:46.474569 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.474520    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:46.474775 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.474617    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:46.483522 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.483480    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq7mn\" (UniqueName: \"kubernetes.io/projected/095b057e-1829-43a4-bcac-4ad5c5e6851d-kube-api-access-hq7mn\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.483710 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.483579    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/095b057e-1829-43a4-bcac-4ad5c5e6851d-openshift-service-ca\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.483710 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.483653    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-maas-api-tls\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.483788 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.483739    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-metrics-certs\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.484451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.484422    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/095b057e-1829-43a4-bcac-4ad5c5e6851d-openshift-service-ca\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.486290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.486268    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-metrics-certs\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.486412 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.486376    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-maas-api-tls\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.505225 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.505190    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq7mn\" (UniqueName: \"kubernetes.io/projected/095b057e-1829-43a4-bcac-4ad5c5e6851d-kube-api-access-hq7mn\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-c24jl\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.627234 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.626837    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:46.983663 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:46.983631    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"]
Nov 02 14:00:46.985721 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:00:46.985691    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095b057e_1829_43a4_bcac_4ad5c5e6851d.slice/crio-45b6ef6488e363182b1106bf5bb09338d2dc98478e12324a18c359a5226f2087 WatchSource:0}: Error finding container 45b6ef6488e363182b1106bf5bb09338d2dc98478e12324a18c359a5226f2087: Status 404 returned error can't find the container with id 45b6ef6488e363182b1106bf5bb09338d2dc98478e12324a18c359a5226f2087
Nov 02 14:00:47.459215 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:47.456147    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:47.459215 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:47.456209    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:47.843959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:47.843195    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl" event={"ID":"095b057e-1829-43a4-bcac-4ad5c5e6851d","Type":"ContainerStarted","Data":"4d5a154e5717e723038af7e9e984e4a96d270f564b4eb792d79c028c969757f9"}
Nov 02 14:00:47.843959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:47.843273    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl" event={"ID":"095b057e-1829-43a4-bcac-4ad5c5e6851d","Type":"ContainerStarted","Data":"45b6ef6488e363182b1106bf5bb09338d2dc98478e12324a18c359a5226f2087"}
Nov 02 14:00:47.843959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:47.843300    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:48.460970 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:48.460426    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:48.460970 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:48.460496    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:49.462960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:49.461432    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:49.462960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:49.461505    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:50.510964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:50.458595    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:50.511664 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:50.511620    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:51.457423 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:51.457379    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:51.457739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:51.457704    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:51.550120 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:51.550069    6284 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:51.781158 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:51.779054    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:51.792982 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:51.791092    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:51.823962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:51.823066    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:51.823962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:51.823139    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.324971 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:52.324043    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.489398 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:52.489358    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:52.506974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:52.503069    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:53.329101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.329045    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.335141 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.335104    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:00:53.373543 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.373501    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:53.373832 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.373801    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.409152 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.409088    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl" podStartSLOduration=6.996552003 podStartE2EDuration="7.40906524s" totalImagesPullingTime="412.513237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:47.90112988 +0000 UTC m=+979.860505488" watchObservedRunningTime="2026-11-02 14:00:53.40906524 +0000 UTC m=+985.368440849"
Nov 02 14:00:53.464540 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.464503    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:53.464828 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.464795    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:53.583033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.581620    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:53.583033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.581695    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.767972 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.760059    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.820487 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.820428    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.884291 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.884177    6284 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:53.884603 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.884556    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.973961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:53.948698    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:00:54.227456 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:54.227324    6284 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.465960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:54.463012    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:54.465960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:54.463079    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:54.604112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:54.604052    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.761106 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:54.761046    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.762328 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:54.762297    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 14:00:54.826774 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:54.826726    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.851601 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:54.837535    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:55.296961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:55.296548    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:55.296961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:55.296623    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:55.582142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:55.582050    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 14:00:55.612882 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:55.612840    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 02 14:00:55.636651 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:55.636608    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:55.637019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:55.636979    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:55.918665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:55.918555    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:00:55.922519 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:55.921121    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:56.248519 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:56.248482    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:00:56.465362 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:56.465307    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 14:00:56.465674 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:56.465642    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 14:00:56.481229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:56.481081    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9zv5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:56.481229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:56.481155    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:57.014904 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.014861    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"]
Nov 02 14:00:57.023497 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.023468    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.033637 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.033605    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"]
Nov 02 14:00:57.111893 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.111849    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 14:00:57.112147 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.111956    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 14:00:57.112147 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.112123    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5wglw\""
Nov 02 14:00:57.112376 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.112197    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 14:00:57.123785 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.123748    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.123997 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.123807    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2njk\" (UniqueName: \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-kube-api-access-x2njk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.123997 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.123851    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.123997 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.123918    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.123997 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.123981    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.124152 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.124020    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.124152 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.124096    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/869b27f2-7068-4501-8d4b-4aae0e159f5e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.236039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.235996    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.236358 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.236329    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.236517 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.236503    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.236720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.236706    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/869b27f2-7068-4501-8d4b-4aae0e159f5e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.236847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.236831    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.237015 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.236999    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2njk\" (UniqueName: \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-kube-api-access-x2njk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.237187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.237174    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.254524 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.254483    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.255919 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.255355    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.257866 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.257768    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.264702 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.264661    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.266892 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.266809    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/869b27f2-7068-4501-8d4b-4aae0e159f5e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.278619 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.276437    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.288860 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.287045    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2njk\" (UniqueName: \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-kube-api-access-x2njk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n885w\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.350407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.350359    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:00:57.521668 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.521585    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:57.523254 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.523232    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:57.524362 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.524338    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:00:57.697013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:57.696049    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"]
Nov 02 14:00:57.705219 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:00:57.705179    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869b27f2_7068_4501_8d4b_4aae0e159f5e.slice/crio-f29aaba35d4c2b0559ca81db59e685f1b454574a9f7366100bda3567bd608cf0 WatchSource:0}: Error finding container f29aaba35d4c2b0559ca81db59e685f1b454574a9f7366100bda3567bd608cf0: Status 404 returned error can't find the container with id f29aaba35d4c2b0559ca81db59e685f1b454574a9f7366100bda3567bd608cf0
Nov 02 14:00:58.489745 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:00:58.489707    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w" event={"ID":"869b27f2-7068-4501-8d4b-4aae0e159f5e","Type":"ContainerStarted","Data":"f29aaba35d4c2b0559ca81db59e685f1b454574a9f7366100bda3567bd608cf0"}
Nov 02 14:01:00.862806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:00.862771    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:01:00.863207 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:00.862836    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:01:01.516424 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:01.516382    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w" event={"ID":"869b27f2-7068-4501-8d4b-4aae0e159f5e","Type":"ContainerStarted","Data":"1490046076bcb7cb42f6588d0deff053a9beca33c4fe23e5c7d50582a20dff02"}
Nov 02 14:01:01.517050 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:01.516975    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:01:01.518711 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:01.518688    6284 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-n885w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:01:01.518883 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:01.518859    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w" podUID="869b27f2-7068-4501-8d4b-4aae0e159f5e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:01:01.579125 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:01.579049    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w" podStartSLOduration=2.425092278 podStartE2EDuration="5.579031226s" totalImagesPullingTime="3.153938948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:01.567783119 +0000 UTC m=+993.527158727" watchObservedRunningTime="2026-11-02 14:01:01.579031226 +0000 UTC m=+993.538406833"
Nov 02 14:01:02.523352 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:02.523320    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:01:14.224561 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:01:14.221526    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:01:14.602304 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:14.595075    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:15.295038 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:15.283062    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 02 14:01:15.643161 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:15.643056    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:16.659114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:16.652106    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:16.659114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:16.652966    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:01:17.236507 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:17.236475    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:01:39.827003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:39.826960    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"]
Nov 02 14:01:39.836150 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:39.836118    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:39.852455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:39.852425    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9eb67b-serving-cert\""
Nov 02 14:01:39.865835 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:39.865797    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"]
Nov 02 14:01:39.992738 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:39.992685    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-openshift-service-ca\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:39.992964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:39.992877    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbb5m\" (UniqueName: \"kubernetes.io/projected/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-kube-api-access-xbb5m\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:39.992964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:39.992952    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-maas-api-tls\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:39.993058 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:39.993012    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-metrics-certs\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:40.094678 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:40.094571    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-metrics-certs\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:40.094879 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:40.094707    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-openshift-service-ca\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:40.094879 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:40.094805    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbb5m\" (UniqueName: \"kubernetes.io/projected/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-kube-api-access-xbb5m\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:40.094879 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:40.094870    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-maas-api-tls\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:40.095499 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:40.095471    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-openshift-service-ca\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:40.097283 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:40.097261    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-metrics-certs\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:40.097395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:40.097308    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-maas-api-tls\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:40.123907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:40.123876    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbb5m\" (UniqueName: \"kubernetes.io/projected/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-kube-api-access-xbb5m\") pod \"maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:40.153416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:40.153369    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:40.479721 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:40.479550    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"]
Nov 02 14:01:40.483869 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:01:40.483838    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87b57ac_f000_4f9b_abcd_184a2b7fffa0.slice/crio-cc0f14cd9079a1615282beb2b673d5ca325625e1739353f092febea83ebb5b3e WatchSource:0}: Error finding container cc0f14cd9079a1615282beb2b673d5ca325625e1739353f092febea83ebb5b3e: Status 404 returned error can't find the container with id cc0f14cd9079a1615282beb2b673d5ca325625e1739353f092febea83ebb5b3e
Nov 02 14:01:41.156820 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:41.145726    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb" event={"ID":"d87b57ac-f000-4f9b-abcd-184a2b7fffa0","Type":"ContainerStarted","Data":"8540741635abfb8faaa88720f203c1a0251155547c1c6ec056e77af1418ed023"}
Nov 02 14:01:41.156820 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:41.145787    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb" event={"ID":"d87b57ac-f000-4f9b-abcd-184a2b7fffa0","Type":"ContainerStarted","Data":"cc0f14cd9079a1615282beb2b673d5ca325625e1739353f092febea83ebb5b3e"}
Nov 02 14:01:41.156820 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:41.146863    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:01:41.190742 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:41.190666    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb" podStartSLOduration=1.8029247339999999 podStartE2EDuration="2.19064413s" totalImagesPullingTime="387.719396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:41.186098442 +0000 UTC m=+1033.145474050" watchObservedRunningTime="2026-11-02 14:01:41.19064413 +0000 UTC m=+1033.150019741"
Nov 02 14:01:43.895896 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:01:43.895853    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:01:48.171485 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:01:48.171457    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:02:13.904378 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:02:13.904339    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:02:20.760596 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:20.760555    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"]
Nov 02 14:02:20.765610 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:20.765535    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:20.792131 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:20.792089    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-981903-serving-cert\""
Nov 02 14:02:20.798019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:20.797975    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"]
Nov 02 14:02:20.950649 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:20.950601    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-metrics-certs\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:20.950848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:20.950736    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-maas-api-tls\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:20.950848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:20.950783    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr24j\" (UniqueName: \"kubernetes.io/projected/2974e7fa-b522-406a-85bc-f412502e8c24-kube-api-access-mr24j\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:20.950986 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:20.950911    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974e7fa-b522-406a-85bc-f412502e8c24-openshift-service-ca\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:21.053099 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.052995    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-maas-api-tls\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:21.053392 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.053366    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr24j\" (UniqueName: \"kubernetes.io/projected/2974e7fa-b522-406a-85bc-f412502e8c24-kube-api-access-mr24j\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:21.065167 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.065124    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974e7fa-b522-406a-85bc-f412502e8c24-openshift-service-ca\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:21.065638 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.065600    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-metrics-certs\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:21.065638 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.063601    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-maas-api-tls\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:21.066746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.066720    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974e7fa-b522-406a-85bc-f412502e8c24-openshift-service-ca\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:21.069142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.069112    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-metrics-certs\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:21.075801 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.075769    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr24j\" (UniqueName: \"kubernetes.io/projected/2974e7fa-b522-406a-85bc-f412502e8c24-kube-api-access-mr24j\") pod \"maas-api-e2e-api-b-981903-7d7cf66864-btkx9\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:21.082732 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.082687    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:21.409520 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.409491    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"]
Nov 02 14:02:21.412200 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:02:21.412163    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2974e7fa_b522_406a_85bc_f412502e8c24.slice/crio-120105f109ad11a625aeb6507c6f1af149fc49ca74efe98372fbd74bbfb52c98 WatchSource:0}: Error finding container 120105f109ad11a625aeb6507c6f1af149fc49ca74efe98372fbd74bbfb52c98: Status 404 returned error can't find the container with id 120105f109ad11a625aeb6507c6f1af149fc49ca74efe98372fbd74bbfb52c98
Nov 02 14:02:21.469400 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:21.469362    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9" event={"ID":"2974e7fa-b522-406a-85bc-f412502e8c24","Type":"ContainerStarted","Data":"120105f109ad11a625aeb6507c6f1af149fc49ca74efe98372fbd74bbfb52c98"}
Nov 02 14:02:22.477763 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:22.477726    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9" event={"ID":"2974e7fa-b522-406a-85bc-f412502e8c24","Type":"ContainerStarted","Data":"4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9"}
Nov 02 14:02:22.478216 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:22.477785    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:22.535013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:22.518517    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9" podStartSLOduration=2.012362879 podStartE2EDuration="2.51849985s" totalImagesPullingTime="506.136971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:22.516273177 +0000 UTC m=+1074.475648785" watchObservedRunningTime="2026-11-02 14:02:22.51849985 +0000 UTC m=+1074.477875457"
Nov 02 14:02:28.491240 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:02:28.491208    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:02:43.899579 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:02:43.899534    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:03:05.897994 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:05.897958    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"]
Nov 02 14:03:05.898803 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:05.898766    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl" podUID="095b057e-1829-43a4-bcac-4ad5c5e6851d" containerName="maas-api" containerID="cri-o://4d5a154e5717e723038af7e9e984e4a96d270f564b4eb792d79c028c969757f9" gracePeriod=30
Nov 02 14:03:06.943962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:06.929813    6284 generic.go:356] "Generic (PLEG): container finished" podID="095b057e-1829-43a4-bcac-4ad5c5e6851d" containerID="4d5a154e5717e723038af7e9e984e4a96d270f564b4eb792d79c028c969757f9" exitCode=0
Nov 02 14:03:06.943962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:06.930076    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl" event={"ID":"095b057e-1829-43a4-bcac-4ad5c5e6851d","Type":"ContainerDied","Data":"4d5a154e5717e723038af7e9e984e4a96d270f564b4eb792d79c028c969757f9"}
Nov 02 14:03:07.466277 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.466188    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:03:07.579241 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.579182    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095b057e-1829-43a4-bcac-4ad5c5e6851d-kube-api-access-hq7mn\" (UniqueName: \"kubernetes.io/projected/095b057e-1829-43a4-bcac-4ad5c5e6851d-kube-api-access-hq7mn\") pod \"095b057e-1829-43a4-bcac-4ad5c5e6851d\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") "
Nov 02 14:03:07.579620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.579595    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095b057e-1829-43a4-bcac-4ad5c5e6851d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/095b057e-1829-43a4-bcac-4ad5c5e6851d-openshift-service-ca\") pod \"095b057e-1829-43a4-bcac-4ad5c5e6851d\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") "
Nov 02 14:03:07.579775 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.579758    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-maas-api-tls\") pod \"095b057e-1829-43a4-bcac-4ad5c5e6851d\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") "
Nov 02 14:03:07.580061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.580043    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-metrics-certs\") pod \"095b057e-1829-43a4-bcac-4ad5c5e6851d\" (UID: \"095b057e-1829-43a4-bcac-4ad5c5e6851d\") "
Nov 02 14:03:07.582918 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.581957    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095b057e-1829-43a4-bcac-4ad5c5e6851d-openshift-service-ca" pod "095b057e-1829-43a4-bcac-4ad5c5e6851d" (UID: "095b057e-1829-43a4-bcac-4ad5c5e6851d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:07.645376 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.645150    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095b057e-1829-43a4-bcac-4ad5c5e6851d-kube-api-access-hq7mn" pod "095b057e-1829-43a4-bcac-4ad5c5e6851d" (UID: "095b057e-1829-43a4-bcac-4ad5c5e6851d"). InnerVolumeSpecName "kube-api-access-hq7mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:07.645376 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.645273    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-metrics-certs" pod "095b057e-1829-43a4-bcac-4ad5c5e6851d" (UID: "095b057e-1829-43a4-bcac-4ad5c5e6851d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:03:07.645376 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.645333    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-maas-api-tls" pod "095b057e-1829-43a4-bcac-4ad5c5e6851d" (UID: "095b057e-1829-43a4-bcac-4ad5c5e6851d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:03:07.684202 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.684097    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:07.684202 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.684136    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq7mn\" (UniqueName: \"kubernetes.io/projected/095b057e-1829-43a4-bcac-4ad5c5e6851d-kube-api-access-hq7mn\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:07.684202 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.684155    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/095b057e-1829-43a4-bcac-4ad5c5e6851d-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:07.684202 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.684171    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/095b057e-1829-43a4-bcac-4ad5c5e6851d-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:07.944780 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.944746    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl" event={"ID":"095b057e-1829-43a4-bcac-4ad5c5e6851d","Type":"ContainerDied","Data":"45b6ef6488e363182b1106bf5bb09338d2dc98478e12324a18c359a5226f2087"}
Nov 02 14:03:07.944780 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.944778    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"
Nov 02 14:03:07.945391 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:07.944822    6284 scope.go:122] "RemoveContainer" containerID="4d5a154e5717e723038af7e9e984e4a96d270f564b4eb792d79c028c969757f9"
Nov 02 14:03:08.033173 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:08.033025    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"]
Nov 02 14:03:08.037431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:08.037378    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-c24jl"]
Nov 02 14:03:08.232200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:08.232069    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095b057e-1829-43a4-bcac-4ad5c5e6851d" path="/var/lib/kubelet/pods/095b057e-1829-43a4-bcac-4ad5c5e6851d/volumes"
Nov 02 14:03:09.686610 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:09.686562    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"]
Nov 02 14:03:09.687204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:09.686845    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb" podUID="d87b57ac-f000-4f9b-abcd-184a2b7fffa0" containerName="maas-api" containerID="cri-o://8540741635abfb8faaa88720f203c1a0251155547c1c6ec056e77af1418ed023" gracePeriod=30
Nov 02 14:03:09.976783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:09.976742    6284 generic.go:356] "Generic (PLEG): container finished" podID="d87b57ac-f000-4f9b-abcd-184a2b7fffa0" containerID="8540741635abfb8faaa88720f203c1a0251155547c1c6ec056e77af1418ed023" exitCode=0
Nov 02 14:03:09.977107 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:09.976787    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb" event={"ID":"d87b57ac-f000-4f9b-abcd-184a2b7fffa0","Type":"ContainerDied","Data":"8540741635abfb8faaa88720f203c1a0251155547c1c6ec056e77af1418ed023"}
Nov 02 14:03:10.011264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.011230    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:03:10.060825 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.060789    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-maas-api-tls\") pod \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") "
Nov 02 14:03:10.061044 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.060915    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-kube-api-access-xbb5m\" (UniqueName: \"kubernetes.io/projected/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-kube-api-access-xbb5m\") pod \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") "
Nov 02 14:03:10.061044 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.060985    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-metrics-certs\") pod \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") "
Nov 02 14:03:10.061450 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.061427    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-openshift-service-ca\") pod \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\" (UID: \"d87b57ac-f000-4f9b-abcd-184a2b7fffa0\") "
Nov 02 14:03:10.062429 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.062400    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-openshift-service-ca" pod "d87b57ac-f000-4f9b-abcd-184a2b7fffa0" (UID: "d87b57ac-f000-4f9b-abcd-184a2b7fffa0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:10.063843 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.063818    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-metrics-certs" pod "d87b57ac-f000-4f9b-abcd-184a2b7fffa0" (UID: "d87b57ac-f000-4f9b-abcd-184a2b7fffa0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:03:10.065428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.065307    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-maas-api-tls" pod "d87b57ac-f000-4f9b-abcd-184a2b7fffa0" (UID: "d87b57ac-f000-4f9b-abcd-184a2b7fffa0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:03:10.065428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.065390    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-kube-api-access-xbb5m" pod "d87b57ac-f000-4f9b-abcd-184a2b7fffa0" (UID: "d87b57ac-f000-4f9b-abcd-184a2b7fffa0"). InnerVolumeSpecName "kube-api-access-xbb5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:10.175978 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.175078    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.175978 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.175126    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.175978 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.175141    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbb5m\" (UniqueName: \"kubernetes.io/projected/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-kube-api-access-xbb5m\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.175978 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.175155    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d87b57ac-f000-4f9b-abcd-184a2b7fffa0-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:10.985611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.985579    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"
Nov 02 14:03:10.985611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.985587    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb" event={"ID":"d87b57ac-f000-4f9b-abcd-184a2b7fffa0","Type":"ContainerDied","Data":"cc0f14cd9079a1615282beb2b673d5ca325625e1739353f092febea83ebb5b3e"}
Nov 02 14:03:10.986164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:10.985669    6284 scope.go:122] "RemoveContainer" containerID="8540741635abfb8faaa88720f203c1a0251155547c1c6ec056e77af1418ed023"
Nov 02 14:03:11.033234 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:11.033195    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"]
Nov 02 14:03:11.042685 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:11.042644    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9eb67b-6b5d8558fd-97ndb"]
Nov 02 14:03:12.171009 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:12.170976    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87b57ac-f000-4f9b-abcd-184a2b7fffa0" path="/var/lib/kubelet/pods/d87b57ac-f000-4f9b-abcd-184a2b7fffa0/volumes"
Nov 02 14:03:13.899859 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:03:13.899815    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:03:26.304247 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.304210    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"]
Nov 02 14:03:26.304862 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.304840    6284 cpu_manager.go:414] "containerMap: removing container" podUID="095b057e-1829-43a4-bcac-4ad5c5e6851d" containerName="maas-api"
Nov 02 14:03:26.304916 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.304865    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095b057e-1829-43a4-bcac-4ad5c5e6851d" containerName="maas-api"
Nov 02 14:03:26.304916 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.304909    6284 cpu_manager.go:414] "containerMap: removing container" podUID="d87b57ac-f000-4f9b-abcd-184a2b7fffa0" containerName="maas-api"
Nov 02 14:03:26.305092 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.304922    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87b57ac-f000-4f9b-abcd-184a2b7fffa0" containerName="maas-api"
Nov 02 14:03:26.305211 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.305195    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87b57ac-f000-4f9b-abcd-184a2b7fffa0" containerName="maas-api"
Nov 02 14:03:26.305251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.305225    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="095b057e-1829-43a4-bcac-4ad5c5e6851d" containerName="maas-api"
Nov 02 14:03:26.310531 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.310508    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.333079 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.333037    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766wd\" (UniqueName: \"kubernetes.io/projected/78a95824-810c-41bd-8518-d4e6a0b99b51-kube-api-access-766wd\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.333275 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.333196    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a95824-810c-41bd-8518-d4e6a0b99b51-openshift-service-ca\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.333275 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.333229    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-metrics-certs\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.333275 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.333252    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-maas-api-tls\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.346285 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.346245    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d776122a-serving-cert\""
Nov 02 14:03:26.351127 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.351089    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"]
Nov 02 14:03:26.434629 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.434588    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766wd\" (UniqueName: \"kubernetes.io/projected/78a95824-810c-41bd-8518-d4e6a0b99b51-kube-api-access-766wd\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.434861 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.434744    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a95824-810c-41bd-8518-d4e6a0b99b51-openshift-service-ca\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.434861 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.434803    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-metrics-certs\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.434861 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.434844    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-maas-api-tls\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.435053 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:03:26.435026    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-serving-cert: secret "maas-api-e2e-mt-d776122a-serving-cert" not found
Nov 02 14:03:26.435120 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:03:26.435107    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-maas-api-tls podName:78a95824-810c-41bd-8518-d4e6a0b99b51 nodeName:}" failed. No retries permitted until 2026-11-02 14:03:26.935086322 +0000 UTC m=+1138.894461918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-maas-api-tls") pod "maas-api-e2e-mt-d776122a-6bbc468f56-w96fs" (UID: "78a95824-810c-41bd-8518-d4e6a0b99b51") : secret "maas-api-e2e-mt-d776122a-serving-cert" not found
Nov 02 14:03:26.435420 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.435401    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a95824-810c-41bd-8518-d4e6a0b99b51-openshift-service-ca\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.437395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.437367    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-metrics-certs\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.474700 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.474667    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766wd\" (UniqueName: \"kubernetes.io/projected/78a95824-810c-41bd-8518-d4e6a0b99b51-kube-api-access-766wd\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.941022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.940919    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-maas-api-tls\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:26.943587 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:26.943553    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-maas-api-tls\") pod \"maas-api-e2e-mt-d776122a-6bbc468f56-w96fs\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:27.227015 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:27.226877    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:27.598996 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:03:27.598914    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a95824_810c_41bd_8518_d4e6a0b99b51.slice/crio-20945a4867368cb4ce5f49b3e0f07599d0b2e1f6f03fcd779d511ef3ac87bd91 WatchSource:0}: Error finding container 20945a4867368cb4ce5f49b3e0f07599d0b2e1f6f03fcd779d511ef3ac87bd91: Status 404 returned error can't find the container with id 20945a4867368cb4ce5f49b3e0f07599d0b2e1f6f03fcd779d511ef3ac87bd91
Nov 02 14:03:27.607798 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:27.607758    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"]
Nov 02 14:03:28.133729 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:28.133691    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs" event={"ID":"78a95824-810c-41bd-8518-d4e6a0b99b51","Type":"ContainerStarted","Data":"20945a4867368cb4ce5f49b3e0f07599d0b2e1f6f03fcd779d511ef3ac87bd91"}
Nov 02 14:03:29.142299 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:29.142260    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs" event={"ID":"78a95824-810c-41bd-8518-d4e6a0b99b51","Type":"ContainerStarted","Data":"e34b9e5ccadcbe031779c13eee4aa81233dbf4f2c7449e15aea1d97c9e93ed19"}
Nov 02 14:03:29.142759 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:29.142338    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:29.183603 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:29.183548    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs" podStartSLOduration=2.717669548 podStartE2EDuration="3.183531355s" totalImagesPullingTime="465.861807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:29.179764887 +0000 UTC m=+1141.139140494" watchObservedRunningTime="2026-11-02 14:03:29.183531355 +0000 UTC m=+1141.142906964"
Nov 02 14:03:35.154904 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:35.154875    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:03:39.081590 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.081548    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"]
Nov 02 14:03:39.087000 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.086970    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.116605 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.116564    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"]
Nov 02 14:03:39.116804 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.116722    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8d5eec-serving-cert\""
Nov 02 14:03:39.224426 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.224377    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-maas-api-tls\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.224426 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.224431    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed241ce8-d3b2-490f-b793-860072f00840-openshift-service-ca\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.224692 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.224565    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v58bm\" (UniqueName: \"kubernetes.io/projected/ed241ce8-d3b2-490f-b793-860072f00840-kube-api-access-v58bm\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.224750 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.224729    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-metrics-certs\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.326243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.326113    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v58bm\" (UniqueName: \"kubernetes.io/projected/ed241ce8-d3b2-490f-b793-860072f00840-kube-api-access-v58bm\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.328028 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.326412    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-metrics-certs\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.328028 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.326691    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-maas-api-tls\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.328028 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.326752    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed241ce8-d3b2-490f-b793-860072f00840-openshift-service-ca\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.329376 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.329342    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed241ce8-d3b2-490f-b793-860072f00840-openshift-service-ca\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.331438 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.331401    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-metrics-certs\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.331743 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.331450    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-maas-api-tls\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.348307 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.348266    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v58bm\" (UniqueName: \"kubernetes.io/projected/ed241ce8-d3b2-490f-b793-860072f00840-kube-api-access-v58bm\") pod \"maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.401434 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.401397    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:39.722849 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:39.722814    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"]
Nov 02 14:03:39.725165 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:03:39.725136    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded241ce8_d3b2_490f_b793_860072f00840.slice/crio-bb13db3f68dd4bb62dec218ad1d0a5aee60284a213fdc625219b36f34a5e8a5a WatchSource:0}: Error finding container bb13db3f68dd4bb62dec218ad1d0a5aee60284a213fdc625219b36f34a5e8a5a: Status 404 returned error can't find the container with id bb13db3f68dd4bb62dec218ad1d0a5aee60284a213fdc625219b36f34a5e8a5a
Nov 02 14:03:40.219623 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:40.219580    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt" event={"ID":"ed241ce8-d3b2-490f-b793-860072f00840","Type":"ContainerStarted","Data":"2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c"}
Nov 02 14:03:40.220102 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:40.219650    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt" event={"ID":"ed241ce8-d3b2-490f-b793-860072f00840","Type":"ContainerStarted","Data":"bb13db3f68dd4bb62dec218ad1d0a5aee60284a213fdc625219b36f34a5e8a5a"}
Nov 02 14:03:40.220102 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:40.219793    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:03:40.664390 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:40.663931    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt" podStartSLOduration=1.3103290109999999 podStartE2EDuration="1.663910502s" totalImagesPullingTime="353.581491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:40.265324021 +0000 UTC m=+1152.224699627" watchObservedRunningTime="2026-11-02 14:03:40.663910502 +0000 UTC m=+1152.623286159"
Nov 02 14:03:40.665440 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:40.665402    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"]
Nov 02 14:03:40.665673 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:40.665648    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9" podUID="2974e7fa-b522-406a-85bc-f412502e8c24" containerName="maas-api" containerID="cri-o://4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9" gracePeriod=30
Nov 02 14:03:41.015154 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.015126    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:03:41.175973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.171203    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-metrics-certs\") pod \"2974e7fa-b522-406a-85bc-f412502e8c24\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") "
Nov 02 14:03:41.175973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.171291    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2974e7fa-b522-406a-85bc-f412502e8c24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974e7fa-b522-406a-85bc-f412502e8c24-openshift-service-ca\") pod \"2974e7fa-b522-406a-85bc-f412502e8c24\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") "
Nov 02 14:03:41.175973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.171339    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-maas-api-tls\") pod \"2974e7fa-b522-406a-85bc-f412502e8c24\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") "
Nov 02 14:03:41.175973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.171457    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2974e7fa-b522-406a-85bc-f412502e8c24-kube-api-access-mr24j\" (UniqueName: \"kubernetes.io/projected/2974e7fa-b522-406a-85bc-f412502e8c24-kube-api-access-mr24j\") pod \"2974e7fa-b522-406a-85bc-f412502e8c24\" (UID: \"2974e7fa-b522-406a-85bc-f412502e8c24\") "
Nov 02 14:03:41.175973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.174307    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2974e7fa-b522-406a-85bc-f412502e8c24-openshift-service-ca" pod "2974e7fa-b522-406a-85bc-f412502e8c24" (UID: "2974e7fa-b522-406a-85bc-f412502e8c24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:41.179983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.176450    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2974e7fa-b522-406a-85bc-f412502e8c24-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:41.197429 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.193464    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2974e7fa-b522-406a-85bc-f412502e8c24-kube-api-access-mr24j" pod "2974e7fa-b522-406a-85bc-f412502e8c24" (UID: "2974e7fa-b522-406a-85bc-f412502e8c24"). InnerVolumeSpecName "kube-api-access-mr24j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:41.197429 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.196448    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-metrics-certs" pod "2974e7fa-b522-406a-85bc-f412502e8c24" (UID: "2974e7fa-b522-406a-85bc-f412502e8c24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:03:41.197429 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.196807    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-maas-api-tls" pod "2974e7fa-b522-406a-85bc-f412502e8c24" (UID: "2974e7fa-b522-406a-85bc-f412502e8c24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:03:41.229762 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.229723    6284 generic.go:356] "Generic (PLEG): container finished" podID="2974e7fa-b522-406a-85bc-f412502e8c24" containerID="4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9" exitCode=0
Nov 02 14:03:41.230330 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.229779    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"
Nov 02 14:03:41.230330 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.229809    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9" event={"ID":"2974e7fa-b522-406a-85bc-f412502e8c24","Type":"ContainerDied","Data":"4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9"}
Nov 02 14:03:41.230330 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.229898    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9" event={"ID":"2974e7fa-b522-406a-85bc-f412502e8c24","Type":"ContainerDied","Data":"120105f109ad11a625aeb6507c6f1af149fc49ca74efe98372fbd74bbfb52c98"}
Nov 02 14:03:41.230330 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.229930    6284 scope.go:122] "RemoveContainer" containerID="4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9"
Nov 02 14:03:41.241557 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.241529    6284 scope.go:122] "RemoveContainer" containerID="4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9"
Nov 02 14:03:41.242004 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:03:41.241980    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9\": container with ID starting with 4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9 not found: ID does not exist" containerID="4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9"
Nov 02 14:03:41.242122 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.242016    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9"} err="failed to get container status \"4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9\": rpc error: code = NotFound desc = could not find container \"4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9\": container with ID starting with 4a9d4773debafc1eb5a78878d886cf9f1dd9737fb9a9610e81e974598da9c5e9 not found: ID does not exist"
Nov 02 14:03:41.281180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.280859    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:41.281180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.280923    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2974e7fa-b522-406a-85bc-f412502e8c24-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:41.281180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.280985    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr24j\" (UniqueName: \"kubernetes.io/projected/2974e7fa-b522-406a-85bc-f412502e8c24-kube-api-access-mr24j\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:03:41.281180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.280998    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"]
Nov 02 14:03:41.290583 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:41.290527    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-981903-7d7cf66864-btkx9"]
Nov 02 14:03:42.198973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:42.192055    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2974e7fa-b522-406a-85bc-f412502e8c24" path="/var/lib/kubelet/pods/2974e7fa-b522-406a-85bc-f412502e8c24/volumes"
Nov 02 14:03:43.897709 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:03:43.897671    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:03:46.237905 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:03:46.237857    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:04:13.898272 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:04:13.898225    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:04:28.380542 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:28.380510    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:04:28.382391 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:28.382360    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:04:34.272803 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.272760    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"]
Nov 02 14:04:34.273615 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.273593    6284 cpu_manager.go:414] "containerMap: removing container" podUID="2974e7fa-b522-406a-85bc-f412502e8c24" containerName="maas-api"
Nov 02 14:04:34.273669 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.273617    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2974e7fa-b522-406a-85bc-f412502e8c24" containerName="maas-api"
Nov 02 14:04:34.273743 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.273733    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="2974e7fa-b522-406a-85bc-f412502e8c24" containerName="maas-api"
Nov 02 14:04:34.278573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.278534    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.301994 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.301911    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e543e0-serving-cert\""
Nov 02 14:04:34.307848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.307807    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"]
Nov 02 14:04:34.356899 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.356849    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.357149 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.357122    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjcbf\" (UniqueName: \"kubernetes.io/projected/fbc71d17-8210-4546-8745-60d22bc32650-kube-api-access-fjcbf\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.357230 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.357207    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-metrics-certs\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.357327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.357313    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc71d17-8210-4546-8745-60d22bc32650-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.458521 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.458481    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjcbf\" (UniqueName: \"kubernetes.io/projected/fbc71d17-8210-4546-8745-60d22bc32650-kube-api-access-fjcbf\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.458731 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.458537    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-metrics-certs\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.458731 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.458580    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc71d17-8210-4546-8745-60d22bc32650-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.458731 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.458664    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.458912 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:04:34.458784    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-serving-cert: secret "maas-api-e2e-ipp-a-e543e0-serving-cert" not found
Nov 02 14:04:34.458912 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:04:34.458851    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-maas-api-tls podName:fbc71d17-8210-4546-8745-60d22bc32650 nodeName:}" failed. No retries permitted until 2026-11-02 14:04:34.958830719 +0000 UTC m=+1206.918206308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-maas-api-tls") pod "maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f" (UID: "fbc71d17-8210-4546-8745-60d22bc32650") : secret "maas-api-e2e-ipp-a-e543e0-serving-cert" not found
Nov 02 14:04:34.459559 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.459526    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc71d17-8210-4546-8745-60d22bc32650-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.462482 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.462447    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-metrics-certs\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.484728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.484697    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjcbf\" (UniqueName: \"kubernetes.io/projected/fbc71d17-8210-4546-8745-60d22bc32650-kube-api-access-fjcbf\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.965669 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.965603    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:34.968493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:34.968458    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:35.198596 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:35.198556    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:35.526016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:35.525978    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"]
Nov 02 14:04:35.747378 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:35.747340    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f" event={"ID":"fbc71d17-8210-4546-8745-60d22bc32650","Type":"ContainerStarted","Data":"6cc953c8aa8695e0fa18fa90afbad574dcfe12171f7d3415b8186f1b833c6195"}
Nov 02 14:04:36.756254 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:36.756200    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f" event={"ID":"fbc71d17-8210-4546-8745-60d22bc32650","Type":"ContainerStarted","Data":"237043630d015248be7a23a1a0ec5e98b33634a3130f3faf9281fd5c07ddf9c6"}
Nov 02 14:04:36.756746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:36.756269    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:36.795006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:36.794948    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f" podStartSLOduration=2.363592203 podStartE2EDuration="2.794917517s" totalImagesPullingTime="431.325314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:36.793651489 +0000 UTC m=+1208.753027095" watchObservedRunningTime="2026-11-02 14:04:36.794917517 +0000 UTC m=+1208.754293178"
Nov 02 14:04:42.771977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:42.771907    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:04:43.899545 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:04:43.899506    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:04:45.900732 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:45.900688    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"]
Nov 02 14:04:45.901281 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:45.900914    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt" podUID="ed241ce8-d3b2-490f-b793-860072f00840" containerName="maas-api" containerID="cri-o://2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c" gracePeriod=30
Nov 02 14:04:46.277959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.275553    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:04:46.449489 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.448171    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed241ce8-d3b2-490f-b793-860072f00840-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed241ce8-d3b2-490f-b793-860072f00840-openshift-service-ca\") pod \"ed241ce8-d3b2-490f-b793-860072f00840\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") "
Nov 02 14:04:46.449489 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.448303    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-maas-api-tls\") pod \"ed241ce8-d3b2-490f-b793-860072f00840\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") "
Nov 02 14:04:46.449489 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.448395    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-metrics-certs\") pod \"ed241ce8-d3b2-490f-b793-860072f00840\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") "
Nov 02 14:04:46.449489 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.448438    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed241ce8-d3b2-490f-b793-860072f00840-kube-api-access-v58bm\" (UniqueName: \"kubernetes.io/projected/ed241ce8-d3b2-490f-b793-860072f00840-kube-api-access-v58bm\") pod \"ed241ce8-d3b2-490f-b793-860072f00840\" (UID: \"ed241ce8-d3b2-490f-b793-860072f00840\") "
Nov 02 14:04:46.450690 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.450658    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed241ce8-d3b2-490f-b793-860072f00840-openshift-service-ca" pod "ed241ce8-d3b2-490f-b793-860072f00840" (UID: "ed241ce8-d3b2-490f-b793-860072f00840"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:46.454325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.454268    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed241ce8-d3b2-490f-b793-860072f00840-kube-api-access-v58bm" pod "ed241ce8-d3b2-490f-b793-860072f00840" (UID: "ed241ce8-d3b2-490f-b793-860072f00840"). InnerVolumeSpecName "kube-api-access-v58bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:46.464169 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.464128    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-maas-api-tls" pod "ed241ce8-d3b2-490f-b793-860072f00840" (UID: "ed241ce8-d3b2-490f-b793-860072f00840"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:04:46.464169 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.464161    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-metrics-certs" pod "ed241ce8-d3b2-490f-b793-860072f00840" (UID: "ed241ce8-d3b2-490f-b793-860072f00840"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:04:46.550701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.550606    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed241ce8-d3b2-490f-b793-860072f00840-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:04:46.550701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.550641    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:04:46.550701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.550650    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed241ce8-d3b2-490f-b793-860072f00840-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:04:46.550701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.550659    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v58bm\" (UniqueName: \"kubernetes.io/projected/ed241ce8-d3b2-490f-b793-860072f00840-kube-api-access-v58bm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:04:46.828344 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.828248    6284 generic.go:356] "Generic (PLEG): container finished" podID="ed241ce8-d3b2-490f-b793-860072f00840" containerID="2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c" exitCode=0
Nov 02 14:04:46.828344 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.828329    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"
Nov 02 14:04:46.828594 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.828327    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt" event={"ID":"ed241ce8-d3b2-490f-b793-860072f00840","Type":"ContainerDied","Data":"2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c"}
Nov 02 14:04:46.828594 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.828471    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt" event={"ID":"ed241ce8-d3b2-490f-b793-860072f00840","Type":"ContainerDied","Data":"bb13db3f68dd4bb62dec218ad1d0a5aee60284a213fdc625219b36f34a5e8a5a"}
Nov 02 14:04:46.828594 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.828494    6284 scope.go:122] "RemoveContainer" containerID="2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c"
Nov 02 14:04:46.841238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.840309    6284 scope.go:122] "RemoveContainer" containerID="2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c"
Nov 02 14:04:46.841238 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:04:46.840751    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c\": container with ID starting with 2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c not found: ID does not exist" containerID="2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c"
Nov 02 14:04:46.841238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.840795    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c"} err="failed to get container status \"2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c\": rpc error: code = NotFound desc = could not find container \"2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c\": container with ID starting with 2f888db1787916c9f9a1fc9334a3a08a2458a633b784051e4a62152c6e2cc43c not found: ID does not exist"
Nov 02 14:04:46.874327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.874291    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"]
Nov 02 14:04:46.878181 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:46.878145    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt"]
Nov 02 14:04:48.170024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:48.169991    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed241ce8-d3b2-490f-b793-860072f00840" path="/var/lib/kubelet/pods/ed241ce8-d3b2-490f-b793-860072f00840/volumes"
Nov 02 14:04:49.240132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:04:49.240087    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8d5eec-69554868b8-nlvvt" podUID="ed241ce8-d3b2-490f-b793-860072f00840" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.78:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:05:05.634960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:05.624593    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"]
Nov 02 14:05:05.634960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:05.625339    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs" podUID="78a95824-810c-41bd-8518-d4e6a0b99b51" containerName="maas-api" containerID="cri-o://e34b9e5ccadcbe031779c13eee4aa81233dbf4f2c7449e15aea1d97c9e93ed19" gracePeriod=30
Nov 02 14:05:06.057087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.057054    6284 generic.go:356] "Generic (PLEG): container finished" podID="78a95824-810c-41bd-8518-d4e6a0b99b51" containerID="e34b9e5ccadcbe031779c13eee4aa81233dbf4f2c7449e15aea1d97c9e93ed19" exitCode=0
Nov 02 14:05:06.057343 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.057320    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs" event={"ID":"78a95824-810c-41bd-8518-d4e6a0b99b51","Type":"ContainerDied","Data":"e34b9e5ccadcbe031779c13eee4aa81233dbf4f2c7449e15aea1d97c9e93ed19"}
Nov 02 14:05:06.057524 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.057379    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs" event={"ID":"78a95824-810c-41bd-8518-d4e6a0b99b51","Type":"ContainerDied","Data":"20945a4867368cb4ce5f49b3e0f07599d0b2e1f6f03fcd779d511ef3ac87bd91"}
Nov 02 14:05:06.057524 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.057445    6284 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20945a4867368cb4ce5f49b3e0f07599d0b2e1f6f03fcd779d511ef3ac87bd91"
Nov 02 14:05:06.058712 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.058689    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:05:06.214224 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.214123    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-metrics-certs\") pod \"78a95824-810c-41bd-8518-d4e6a0b99b51\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") "
Nov 02 14:05:06.214224 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.214203    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-maas-api-tls\") pod \"78a95824-810c-41bd-8518-d4e6a0b99b51\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") "
Nov 02 14:05:06.214485 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.214333    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a95824-810c-41bd-8518-d4e6a0b99b51-kube-api-access-766wd\" (UniqueName: \"kubernetes.io/projected/78a95824-810c-41bd-8518-d4e6a0b99b51-kube-api-access-766wd\") pod \"78a95824-810c-41bd-8518-d4e6a0b99b51\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") "
Nov 02 14:05:06.214485 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.214388    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a95824-810c-41bd-8518-d4e6a0b99b51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a95824-810c-41bd-8518-d4e6a0b99b51-openshift-service-ca\") pod \"78a95824-810c-41bd-8518-d4e6a0b99b51\" (UID: \"78a95824-810c-41bd-8518-d4e6a0b99b51\") "
Nov 02 14:05:06.215491 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.215459    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a95824-810c-41bd-8518-d4e6a0b99b51-openshift-service-ca" pod "78a95824-810c-41bd-8518-d4e6a0b99b51" (UID: "78a95824-810c-41bd-8518-d4e6a0b99b51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:05:06.218622 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.218589    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a95824-810c-41bd-8518-d4e6a0b99b51-kube-api-access-766wd" pod "78a95824-810c-41bd-8518-d4e6a0b99b51" (UID: "78a95824-810c-41bd-8518-d4e6a0b99b51"). InnerVolumeSpecName "kube-api-access-766wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:05:06.218868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.218846    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-maas-api-tls" pod "78a95824-810c-41bd-8518-d4e6a0b99b51" (UID: "78a95824-810c-41bd-8518-d4e6a0b99b51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:05:06.219520 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.219497    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-metrics-certs" pod "78a95824-810c-41bd-8518-d4e6a0b99b51" (UID: "78a95824-810c-41bd-8518-d4e6a0b99b51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:05:06.320277 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.320236    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:05:06.328058 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.328022    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78a95824-810c-41bd-8518-d4e6a0b99b51-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:05:06.328299 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.328281    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766wd\" (UniqueName: \"kubernetes.io/projected/78a95824-810c-41bd-8518-d4e6a0b99b51-kube-api-access-766wd\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:05:06.328433 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:06.328420    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a95824-810c-41bd-8518-d4e6a0b99b51-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:05:07.065924 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:07.065877    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"
Nov 02 14:05:07.117754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:07.117718    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"]
Nov 02 14:05:07.126890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:07.126855    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d776122a-6bbc468f56-w96fs"]
Nov 02 14:05:08.172459 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.172420    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a95824-810c-41bd-8518-d4e6a0b99b51" path="/var/lib/kubelet/pods/78a95824-810c-41bd-8518-d4e6a0b99b51/volumes"
Nov 02 14:05:08.758719 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.758676    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"]
Nov 02 14:05:08.759325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.759309    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ed241ce8-d3b2-490f-b793-860072f00840" containerName="maas-api"
Nov 02 14:05:08.759325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.759327    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed241ce8-d3b2-490f-b793-860072f00840" containerName="maas-api"
Nov 02 14:05:08.759440 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.759340    6284 cpu_manager.go:414] "containerMap: removing container" podUID="78a95824-810c-41bd-8518-d4e6a0b99b51" containerName="maas-api"
Nov 02 14:05:08.759440 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.759345    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a95824-810c-41bd-8518-d4e6a0b99b51" containerName="maas-api"
Nov 02 14:05:08.759440 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.759428    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a95824-810c-41bd-8518-d4e6a0b99b51" containerName="maas-api"
Nov 02 14:05:08.759535 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.759445    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed241ce8-d3b2-490f-b793-860072f00840" containerName="maas-api"
Nov 02 14:05:08.764331 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.764305    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:05:08.776701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.776666    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0b8980-aff0-4e4d-873e-a3a0d17468b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf\" (UID: \"da0b8980-aff0-4e4d-873e-a3a0d17468b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:05:08.776868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.776723    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd47h\" (UniqueName: \"kubernetes.io/projected/da0b8980-aff0-4e4d-873e-a3a0d17468b1-kube-api-access-kd47h\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf\" (UID: \"da0b8980-aff0-4e4d-873e-a3a0d17468b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:05:08.788066 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.788032    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e543e0-dockercfg-j2wx4\""
Nov 02 14:05:08.788066 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.788054    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e543e0\""
Nov 02 14:05:08.792279 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.792235    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"]
Nov 02 14:05:08.883495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.879680    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0b8980-aff0-4e4d-873e-a3a0d17468b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf\" (UID: \"da0b8980-aff0-4e4d-873e-a3a0d17468b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:05:08.883495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.879753    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd47h\" (UniqueName: \"kubernetes.io/projected/da0b8980-aff0-4e4d-873e-a3a0d17468b1-kube-api-access-kd47h\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf\" (UID: \"da0b8980-aff0-4e4d-873e-a3a0d17468b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:05:08.883495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.880607    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0b8980-aff0-4e4d-873e-a3a0d17468b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf\" (UID: \"da0b8980-aff0-4e4d-873e-a3a0d17468b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:05:08.901791 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.901756    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"]
Nov 02 14:05:08.939446 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.939359    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd47h\" (UniqueName: \"kubernetes.io/projected/da0b8980-aff0-4e4d-873e-a3a0d17468b1-kube-api-access-kd47h\") pod \"payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf\" (UID: \"da0b8980-aff0-4e4d-873e-a3a0d17468b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:05:08.943317 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.943278    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"]
Nov 02 14:05:08.943501 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.943422    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:05:08.987870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.987834    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89c53ef-367a-4e28-853e-aea9d487c9d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485\" (UID: \"f89c53ef-367a-4e28-853e-aea9d487c9d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:05:08.988247 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:08.988217    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw9j8\" (UniqueName: \"kubernetes.io/projected/f89c53ef-367a-4e28-853e-aea9d487c9d4-kube-api-access-lw9j8\") pod \"payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485\" (UID: \"f89c53ef-367a-4e28-853e-aea9d487c9d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:05:09.079502 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:09.079409    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:05:09.089383 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:09.089352    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89c53ef-367a-4e28-853e-aea9d487c9d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485\" (UID: \"f89c53ef-367a-4e28-853e-aea9d487c9d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:05:09.089608 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:09.089585    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9j8\" (UniqueName: \"kubernetes.io/projected/f89c53ef-367a-4e28-853e-aea9d487c9d4-kube-api-access-lw9j8\") pod \"payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485\" (UID: \"f89c53ef-367a-4e28-853e-aea9d487c9d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:05:09.090294 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:09.090265    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89c53ef-367a-4e28-853e-aea9d487c9d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485\" (UID: \"f89c53ef-367a-4e28-853e-aea9d487c9d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:05:09.111073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:09.111042    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw9j8\" (UniqueName: \"kubernetes.io/projected/f89c53ef-367a-4e28-853e-aea9d487c9d4-kube-api-access-lw9j8\") pod \"payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485\" (UID: \"f89c53ef-367a-4e28-853e-aea9d487c9d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:05:09.275172 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:09.275127    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:05:09.411021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:09.410985    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"]
Nov 02 14:05:09.411990 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:05:09.411163    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0b8980_aff0_4e4d_873e_a3a0d17468b1.slice/crio-862b1470296f071b088bc4b190fe795cc63ce5aa2f6f2d733c35aaeb303ccfc4 WatchSource:0}: Error finding container 862b1470296f071b088bc4b190fe795cc63ce5aa2f6f2d733c35aaeb303ccfc4: Status 404 returned error can't find the container with id 862b1470296f071b088bc4b190fe795cc63ce5aa2f6f2d733c35aaeb303ccfc4
Nov 02 14:05:09.414947 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:09.414909    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:05:09.593482 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:09.593454    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"]
Nov 02 14:05:09.595433 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:05:09.595399    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89c53ef_367a_4e28_853e_aea9d487c9d4.slice/crio-33fff3621b1dfce07d069948e7ec9d043749a71a1a231219bc21968180306db7 WatchSource:0}: Error finding container 33fff3621b1dfce07d069948e7ec9d043749a71a1a231219bc21968180306db7: Status 404 returned error can't find the container with id 33fff3621b1dfce07d069948e7ec9d043749a71a1a231219bc21968180306db7
Nov 02 14:05:10.196046 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:10.191775    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf" event={"ID":"da0b8980-aff0-4e4d-873e-a3a0d17468b1","Type":"ContainerStarted","Data":"862b1470296f071b088bc4b190fe795cc63ce5aa2f6f2d733c35aaeb303ccfc4"}
Nov 02 14:05:10.196046 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:10.191871    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485" event={"ID":"f89c53ef-367a-4e28-853e-aea9d487c9d4","Type":"ContainerStarted","Data":"33fff3621b1dfce07d069948e7ec9d043749a71a1a231219bc21968180306db7"}
Nov 02 14:05:12.208575 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:12.208536    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf" event={"ID":"da0b8980-aff0-4e4d-873e-a3a0d17468b1","Type":"ContainerStarted","Data":"1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691"}
Nov 02 14:05:12.209068 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:12.208603    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:05:12.210261 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:12.210233    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485" event={"ID":"f89c53ef-367a-4e28-853e-aea9d487c9d4","Type":"ContainerStarted","Data":"48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274"}
Nov 02 14:05:12.210387 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:12.210327    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:05:12.247565 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:12.247496    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf" podStartSLOduration=1.902322087 podStartE2EDuration="4.247475309s" totalImagesPullingTime="2.345153222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:12.2453229 +0000 UTC m=+1244.204698508" watchObservedRunningTime="2026-11-02 14:05:12.247475309 +0000 UTC m=+1244.206850919"
Nov 02 14:05:12.283076 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:12.282991    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485" podStartSLOduration=2.160748947 podStartE2EDuration="4.28294546s" totalImagesPullingTime="2.122196513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:12.281281746 +0000 UTC m=+1244.240657366" watchObservedRunningTime="2026-11-02 14:05:12.28294546 +0000 UTC m=+1244.242321059"
Nov 02 14:05:13.897322 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:05:13.897279    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:05:18.027860 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:18.027176    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"]
Nov 02 14:05:18.038466 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:18.038426    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efe3c946-l2gzk"]
Nov 02 14:05:18.173760 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:18.173719    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b31a177-eabd-42ff-8ffb-9044c2f936f9" path="/var/lib/kubelet/pods/1b31a177-eabd-42ff-8ffb-9044c2f936f9/volumes"
Nov 02 14:05:20.563434 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.563393    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"]
Nov 02 14:05:20.571821 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.571782    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:05:20.593960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.592790    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"]
Nov 02 14:05:20.593960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.593785    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ae366a-dockercfg-2dhnb\""
Nov 02 14:05:20.593960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.593946    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ae366a\""
Nov 02 14:05:20.638804 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.638752    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48a87e-108d-40f7-8a9e-b852edc482f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq\" (UID: \"dc48a87e-108d-40f7-8a9e-b852edc482f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:05:20.639195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.638870    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btfff\" (UniqueName: \"kubernetes.io/projected/dc48a87e-108d-40f7-8a9e-b852edc482f1-kube-api-access-btfff\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq\" (UID: \"dc48a87e-108d-40f7-8a9e-b852edc482f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:05:20.741629 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.741579    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btfff\" (UniqueName: \"kubernetes.io/projected/dc48a87e-108d-40f7-8a9e-b852edc482f1-kube-api-access-btfff\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq\" (UID: \"dc48a87e-108d-40f7-8a9e-b852edc482f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:05:20.741883 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.741855    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48a87e-108d-40f7-8a9e-b852edc482f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq\" (UID: \"dc48a87e-108d-40f7-8a9e-b852edc482f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:05:20.742763 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.742741    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48a87e-108d-40f7-8a9e-b852edc482f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq\" (UID: \"dc48a87e-108d-40f7-8a9e-b852edc482f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:05:20.764337 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.764252    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btfff\" (UniqueName: \"kubernetes.io/projected/dc48a87e-108d-40f7-8a9e-b852edc482f1-kube-api-access-btfff\") pod \"payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq\" (UID: \"dc48a87e-108d-40f7-8a9e-b852edc482f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:05:20.934735 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:20.934638    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:05:21.267256 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:21.267220    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"]
Nov 02 14:05:21.269760 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:05:21.269722    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc48a87e_108d_40f7_8a9e_b852edc482f1.slice/crio-eaf0a1ddddd5cf5fc7c8cc6fe1a815eec0a0054cda138c00d4bf47175ef89533 WatchSource:0}: Error finding container eaf0a1ddddd5cf5fc7c8cc6fe1a815eec0a0054cda138c00d4bf47175ef89533: Status 404 returned error can't find the container with id eaf0a1ddddd5cf5fc7c8cc6fe1a815eec0a0054cda138c00d4bf47175ef89533
Nov 02 14:05:21.305957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:21.305051    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq" event={"ID":"dc48a87e-108d-40f7-8a9e-b852edc482f1","Type":"ContainerStarted","Data":"eaf0a1ddddd5cf5fc7c8cc6fe1a815eec0a0054cda138c00d4bf47175ef89533"}
Nov 02 14:05:22.316010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:22.315970    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq" event={"ID":"dc48a87e-108d-40f7-8a9e-b852edc482f1","Type":"ContainerStarted","Data":"9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621"}
Nov 02 14:05:22.316450 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:22.316051    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:05:22.359431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:22.359345    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq" podStartSLOduration=2.359330822 podStartE2EDuration="2.359330822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:22.357882665 +0000 UTC m=+1254.317258272" watchObservedRunningTime="2026-11-02 14:05:22.359330822 +0000 UTC m=+1254.318706429"
Nov 02 14:05:23.220168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:23.220122    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:05:23.221102 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:23.220386    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:05:33.327424 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:33.327394    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:05:38.222345 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.222306    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"]
Nov 02 14:05:38.227754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.227726    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.247042 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.247005    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c7d2df-serving-cert\""
Nov 02 14:05:38.262606 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.262573    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"]
Nov 02 14:05:38.307091 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.307040    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-metrics-certs\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.307310 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.307129    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-openshift-service-ca\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.307310 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.307241    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-maas-api-tls\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.307442 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.307307    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4s8\" (UniqueName: \"kubernetes.io/projected/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-kube-api-access-rq4s8\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.409191 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.409145    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-metrics-certs\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.409408 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.409214    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-openshift-service-ca\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.409408 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.409307    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-maas-api-tls\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.409408 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.409364    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4s8\" (UniqueName: \"kubernetes.io/projected/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-kube-api-access-rq4s8\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.410003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.409978    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-openshift-service-ca\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.411701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.411680    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-metrics-certs\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.411758 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.411724    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-maas-api-tls\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.431816 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.431776    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4s8\" (UniqueName: \"kubernetes.io/projected/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-kube-api-access-rq4s8\") pod \"maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.545618 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.545581    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:38.871415 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:38.871387    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"]
Nov 02 14:05:38.874665 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:05:38.874634    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ceefcc_f51c_4e09_ac59_b3b1008e75f6.slice/crio-5fafdd3f26aac45783731d72b1203890e1e9fdb3ad23d9241eb4c83c83307c23 WatchSource:0}: Error finding container 5fafdd3f26aac45783731d72b1203890e1e9fdb3ad23d9241eb4c83c83307c23: Status 404 returned error can't find the container with id 5fafdd3f26aac45783731d72b1203890e1e9fdb3ad23d9241eb4c83c83307c23
Nov 02 14:05:39.450676 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:39.450626    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l" event={"ID":"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6","Type":"ContainerStarted","Data":"6c51c0553ea6af3e3daa9d22656120ed4d94a91a5c241a93ad0546d860ac2e61"}
Nov 02 14:05:39.451146 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:39.450706    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:39.451146 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:39.450729    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l" event={"ID":"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6","Type":"ContainerStarted","Data":"5fafdd3f26aac45783731d72b1203890e1e9fdb3ad23d9241eb4c83c83307c23"}
Nov 02 14:05:39.490779 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:39.490712    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l" podStartSLOduration=1.080784073 podStartE2EDuration="1.49069164s" totalImagesPullingTime="409.907567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:39.489612612 +0000 UTC m=+1271.448988217" watchObservedRunningTime="2026-11-02 14:05:39.49069164 +0000 UTC m=+1271.450067250"
Nov 02 14:05:43.898723 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:05:43.898679    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:05:45.466088 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:45.466058    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:05:50.058289 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.058253    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"]
Nov 02 14:05:50.065129 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.065099    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.088055 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.088020    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-fb206f-serving-cert\""
Nov 02 14:05:50.092121 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.092084    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"]
Nov 02 14:05:50.212788 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.212743    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbf4\" (UniqueName: \"kubernetes.io/projected/43bf93e3-524e-4165-ba2b-3d87d7855005-kube-api-access-njbf4\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.213076 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.212902    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.213076 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.213021    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-metrics-certs\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.213240 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.213105    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43bf93e3-524e-4165-ba2b-3d87d7855005-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.314999 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.314847    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbf4\" (UniqueName: \"kubernetes.io/projected/43bf93e3-524e-4165-ba2b-3d87d7855005-kube-api-access-njbf4\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.315222 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.315084    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.315222 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.315126    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-metrics-certs\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.315361 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.315228    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43bf93e3-524e-4165-ba2b-3d87d7855005-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.316062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.316031    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43bf93e3-524e-4165-ba2b-3d87d7855005-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.318330 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.318292    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.318667 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.318628    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-metrics-certs\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.337212 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.337181    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbf4\" (UniqueName: \"kubernetes.io/projected/43bf93e3-524e-4165-ba2b-3d87d7855005-kube-api-access-njbf4\") pod \"maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.382554 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.382516    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:50.699122 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:50.699089    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"]
Nov 02 14:05:50.701219 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:05:50.701182    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bf93e3_524e_4165_ba2b_3d87d7855005.slice/crio-9baa580c332c4ee0e69db30f31985d77c2edcf77ec2c8e566cdf86c77bebeada WatchSource:0}: Error finding container 9baa580c332c4ee0e69db30f31985d77c2edcf77ec2c8e566cdf86c77bebeada: Status 404 returned error can't find the container with id 9baa580c332c4ee0e69db30f31985d77c2edcf77ec2c8e566cdf86c77bebeada
Nov 02 14:05:51.553271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:51.553237    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q" event={"ID":"43bf93e3-524e-4165-ba2b-3d87d7855005","Type":"ContainerStarted","Data":"96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb"}
Nov 02 14:05:51.553751 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:51.553289    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q" event={"ID":"43bf93e3-524e-4165-ba2b-3d87d7855005","Type":"ContainerStarted","Data":"9baa580c332c4ee0e69db30f31985d77c2edcf77ec2c8e566cdf86c77bebeada"}
Nov 02 14:05:51.553751 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:51.553420    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:05:51.594843 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:51.594769    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q" podStartSLOduration=1.276848007 podStartE2EDuration="1.594733676s" totalImagesPullingTime="317.885669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:51.592039381 +0000 UTC m=+1283.551414981" watchObservedRunningTime="2026-11-02 14:05:51.594733676 +0000 UTC m=+1283.554109285"
Nov 02 14:05:57.576481 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:05:57.576427    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q" podUID="43bf93e3-524e-4165-ba2b-3d87d7855005" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.84:8443/health\": dial tcp 10.134.0.84:8443: connect: connection refused"
Nov 02 14:06:00.040226 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:00.040182    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"]
Nov 02 14:06:00.040711 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:00.040427    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy" containerID="cri-o://f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da" gracePeriod=30
Nov 02 14:06:01.643830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:01.643799    6284 generic.go:356] "Generic (PLEG): container finished" podID="43bf93e3-524e-4165-ba2b-3d87d7855005" containerID="96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb" exitCode=1
Nov 02 14:06:01.644376 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:01.643879    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q" event={"ID":"43bf93e3-524e-4165-ba2b-3d87d7855005","Type":"ContainerDied","Data":"96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb"}
Nov 02 14:06:01.644446 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:01.644412    6284 scope.go:122] "RemoveContainer" containerID="96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb"
Nov 02 14:06:02.653331 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:02.653296    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q" event={"ID":"43bf93e3-524e-4165-ba2b-3d87d7855005","Type":"ContainerStarted","Data":"5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031"}
Nov 02 14:06:02.653812 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:02.653511    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:06:05.331704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.331673    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:06:05.490426 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.490387    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-socket\") pod \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") "
Nov 02 14:06:05.490640 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.490515    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-envoy\") pod \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") "
Nov 02 14:06:05.490640 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.490584    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-token\" (UniqueName: \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-token\") pod \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") "
Nov 02 14:06:05.490640 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.490613    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-kube-api-access-nv27v\" (UniqueName: \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-kube-api-access-nv27v\") pod \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") "
Nov 02 14:06:05.490951 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.490639    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-credential-socket\") pod \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") "
Nov 02 14:06:05.490951 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.490679    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istiod-ca-cert\") pod \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") "
Nov 02 14:06:05.490951 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.490720    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-socket" pod "02c9ecb9-4cd3-432d-8f15-a7af2719a972" (UID: "02c9ecb9-4cd3-432d-8f15-a7af2719a972"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:05.490951 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.490879    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-data\") pod \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") "
Nov 02 14:06:05.491163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.490957    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-certs\") pod \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") "
Nov 02 14:06:05.491163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.491037    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-podinfo\") pod \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\" (UID: \"02c9ecb9-4cd3-432d-8f15-a7af2719a972\") "
Nov 02 14:06:05.491163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.491039    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-credential-socket" pod "02c9ecb9-4cd3-432d-8f15-a7af2719a972" (UID: "02c9ecb9-4cd3-432d-8f15-a7af2719a972"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:05.491341 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.491283    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-certs" pod "02c9ecb9-4cd3-432d-8f15-a7af2719a972" (UID: "02c9ecb9-4cd3-432d-8f15-a7af2719a972"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:05.491341 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.491302    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-data" pod "02c9ecb9-4cd3-432d-8f15-a7af2719a972" (UID: "02c9ecb9-4cd3-432d-8f15-a7af2719a972"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:05.491432 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.491365    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istiod-ca-cert" pod "02c9ecb9-4cd3-432d-8f15-a7af2719a972" (UID: "02c9ecb9-4cd3-432d-8f15-a7af2719a972"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:05.491766 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.491742    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:05.491810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.491775    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:05.491810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.491792    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:05.491810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.491806    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:05.491979 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.491819    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:05.493343 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.493314    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-envoy" pod "02c9ecb9-4cd3-432d-8f15-a7af2719a972" (UID: "02c9ecb9-4cd3-432d-8f15-a7af2719a972"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:05.493504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.493484    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-podinfo" pod "02c9ecb9-4cd3-432d-8f15-a7af2719a972" (UID: "02c9ecb9-4cd3-432d-8f15-a7af2719a972"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:06:05.493975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.493909    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-token" pod "02c9ecb9-4cd3-432d-8f15-a7af2719a972" (UID: "02c9ecb9-4cd3-432d-8f15-a7af2719a972"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:05.493975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.493924    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-kube-api-access-nv27v" pod "02c9ecb9-4cd3-432d-8f15-a7af2719a972" (UID: "02c9ecb9-4cd3-432d-8f15-a7af2719a972"). InnerVolumeSpecName "kube-api-access-nv27v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:05.594588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.594495    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:05.594588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.594525    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:05.594588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.594538    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv27v\" (UniqueName: \"kubernetes.io/projected/02c9ecb9-4cd3-432d-8f15-a7af2719a972-kube-api-access-nv27v\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:05.594588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.594551    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c9ecb9-4cd3-432d-8f15-a7af2719a972-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:05.678265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.678220    6284 generic.go:356] "Generic (PLEG): container finished" podID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerID="f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da" exitCode=0
Nov 02 14:06:05.678480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.678316    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" event={"ID":"02c9ecb9-4cd3-432d-8f15-a7af2719a972","Type":"ContainerDied","Data":"f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da"}
Nov 02 14:06:05.678480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.678332    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"
Nov 02 14:06:05.678480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.678373    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn" event={"ID":"02c9ecb9-4cd3-432d-8f15-a7af2719a972","Type":"ContainerDied","Data":"36c7500e9413bb279a0045d65bab44133b009ea4fdb747e7ebec2e7e17af982d"}
Nov 02 14:06:05.678480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.678399    6284 scope.go:122] "RemoveContainer" containerID="f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da"
Nov 02 14:06:05.697040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.696818    6284 scope.go:122] "RemoveContainer" containerID="f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da"
Nov 02 14:06:05.697372 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:06:05.697338    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da\": container with ID starting with f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da not found: ID does not exist" containerID="f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da"
Nov 02 14:06:05.697504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.697385    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da"} err="failed to get container status \"f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da\": rpc error: code = NotFound desc = could not find container \"f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da\": container with ID starting with f6df27370b0b79ae7220a239c2c4143b5afc59e2730cdde1039d1f44600f65da not found: ID does not exist"
Nov 02 14:06:05.727463 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.727381    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"]
Nov 02 14:06:05.734324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.734286    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-zgzpn"]
Nov 02 14:06:05.809346 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.809301    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"]
Nov 02 14:06:05.810331 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.810299    6284 cpu_manager.go:414] "containerMap: removing container" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy"
Nov 02 14:06:05.810331 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.810325    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy"
Nov 02 14:06:05.810572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.810473    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" containerName="istio-proxy"
Nov 02 14:06:05.816189 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.816160    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:06:05.847108 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.844157    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-17a7ba\""
Nov 02 14:06:05.847108 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.845293    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-17a7ba-dockercfg-d998c\""
Nov 02 14:06:05.868114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:05.868080    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"]
Nov 02 14:06:06.000910 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.000863    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5\" (UID: \"75b32100-8dd4-4cbe-8aae-4d96d397fa8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:06:06.001134 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.001048    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5849\" (UniqueName: \"kubernetes.io/projected/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-kube-api-access-g5849\") pod \"payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5\" (UID: \"75b32100-8dd4-4cbe-8aae-4d96d397fa8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:06:06.102533 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.102410    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5\" (UID: \"75b32100-8dd4-4cbe-8aae-4d96d397fa8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:06:06.102739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.102536    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5849\" (UniqueName: \"kubernetes.io/projected/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-kube-api-access-g5849\") pod \"payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5\" (UID: \"75b32100-8dd4-4cbe-8aae-4d96d397fa8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:06:06.103537 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.103494    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5\" (UID: \"75b32100-8dd4-4cbe-8aae-4d96d397fa8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:06:06.124913 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.124875    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5849\" (UniqueName: \"kubernetes.io/projected/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-kube-api-access-g5849\") pod \"payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5\" (UID: \"75b32100-8dd4-4cbe-8aae-4d96d397fa8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:06:06.152610 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.152566    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:06:06.174267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.174239    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c9ecb9-4cd3-432d-8f15-a7af2719a972" path="/var/lib/kubelet/pods/02c9ecb9-4cd3-432d-8f15-a7af2719a972/volumes"
Nov 02 14:06:06.478568 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.478539    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"]
Nov 02 14:06:06.481164 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:06:06.481136    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b32100_8dd4_4cbe_8aae_4d96d397fa8e.slice/crio-e9537750bb6f4e0580b6a76af549fa45ea3113154dea86011b4b65ed1083f9cc WatchSource:0}: Error finding container e9537750bb6f4e0580b6a76af549fa45ea3113154dea86011b4b65ed1083f9cc: Status 404 returned error can't find the container with id e9537750bb6f4e0580b6a76af549fa45ea3113154dea86011b4b65ed1083f9cc
Nov 02 14:06:06.692307 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.692260    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5" event={"ID":"75b32100-8dd4-4cbe-8aae-4d96d397fa8e","Type":"ContainerStarted","Data":"c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e"}
Nov 02 14:06:06.692534 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.692342    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5" event={"ID":"75b32100-8dd4-4cbe-8aae-4d96d397fa8e","Type":"ContainerStarted","Data":"e9537750bb6f4e0580b6a76af549fa45ea3113154dea86011b4b65ed1083f9cc"}
Nov 02 14:06:06.693045 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.693014    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:06:06.730325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:06.730217    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5" podStartSLOduration=1.7302023 podStartE2EDuration="1.7302023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:06.728774162 +0000 UTC m=+1298.688149769" watchObservedRunningTime="2026-11-02 14:06:06.7302023 +0000 UTC m=+1298.689577942"
Nov 02 14:06:08.667409 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.667374    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:06:08.682747 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.682711    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"]
Nov 02 14:06:08.687868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.687845    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.709194 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.709159    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"]
Nov 02 14:06:08.733334 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.733301    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-60ad62-serving-cert\""
Nov 02 14:06:08.743466 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.743397    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.743677 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.743539    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crftl\" (UniqueName: \"kubernetes.io/projected/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-kube-api-access-crftl\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.743677 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.743644    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.743836 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.743809    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-metrics-certs\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.846155 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.846107    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.846367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.846170    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-metrics-certs\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.846367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.846355    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.846492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.846477    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crftl\" (UniqueName: \"kubernetes.io/projected/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-kube-api-access-crftl\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.847559 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.847527    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.850333 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.850287    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-metrics-certs\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.850555 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.850313    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-maas-api-tls\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:08.871559 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:08.871523    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crftl\" (UniqueName: \"kubernetes.io/projected/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-kube-api-access-crftl\") pod \"maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:09.003588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:09.003551    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:09.471373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:09.471335    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"]
Nov 02 14:06:09.474641 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:06:09.474598    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3dc892f_bfcc_46da_9f5a_a004aa8b2931.slice/crio-d49c39d5bbe7bfe5fee832bc20014d8d45eabb7c09094a3e9795dcfa03e2e5dd WatchSource:0}: Error finding container d49c39d5bbe7bfe5fee832bc20014d8d45eabb7c09094a3e9795dcfa03e2e5dd: Status 404 returned error can't find the container with id d49c39d5bbe7bfe5fee832bc20014d8d45eabb7c09094a3e9795dcfa03e2e5dd
Nov 02 14:06:09.725397 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:09.725301    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689" event={"ID":"f3dc892f-bfcc-46da-9f5a-a004aa8b2931","Type":"ContainerStarted","Data":"d49c39d5bbe7bfe5fee832bc20014d8d45eabb7c09094a3e9795dcfa03e2e5dd"}
Nov 02 14:06:10.735217 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:10.735177    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689" event={"ID":"f3dc892f-bfcc-46da-9f5a-a004aa8b2931","Type":"ContainerStarted","Data":"a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988"}
Nov 02 14:06:10.735640 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:10.735261    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:10.776512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:10.776440    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689" podStartSLOduration=2.459974941 podStartE2EDuration="2.776417955s" totalImagesPullingTime="316.443014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:10.771771704 +0000 UTC m=+1302.731147318" watchObservedRunningTime="2026-11-02 14:06:10.776417955 +0000 UTC m=+1302.735793562"
Nov 02 14:06:13.895789 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:06:13.895747    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:06:16.749288 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:16.749259    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:06:17.703472 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:17.703441    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:06:29.112196 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:29.112162    6284 scope.go:122] "RemoveContainer" containerID="e8bc71ff37577a65deefa0d0f7cb83f01b7aa33a31bd0853658452a9bf413ed4"
Nov 02 14:06:29.175199 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:29.175144    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"]
Nov 02 14:06:29.175420 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:29.175400    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy" containerID="cri-o://477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906" gracePeriod=30
Nov 02 14:06:34.472112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.472078    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 14:06:34.584727 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.584685    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-podinfo\") pod \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") "
Nov 02 14:06:34.584948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.584744    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-token\" (UniqueName: \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-token\") pod \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") "
Nov 02 14:06:34.584948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.584821    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-kube-api-access-pkdm2\" (UniqueName: \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-kube-api-access-pkdm2\") pod \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") "
Nov 02 14:06:34.584948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.584858    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istiod-ca-cert\") pod \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") "
Nov 02 14:06:34.584948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.584916    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-data\") pod \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") "
Nov 02 14:06:34.585169 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.584961    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-credential-socket\") pod \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") "
Nov 02 14:06:34.585169 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.585034    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-envoy\") pod \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") "
Nov 02 14:06:34.585269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.585258    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-certs\") pod \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") "
Nov 02 14:06:34.585324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.585293    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-socket\") pod \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\" (UID: \"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e\") "
Nov 02 14:06:34.586375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.585480    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istiod-ca-cert" pod "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" (UID: "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:34.586375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.585720    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-socket" pod "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" (UID: "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:34.586375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.585921    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-certs" pod "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" (UID: "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:34.586375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.586054    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-credential-socket" pod "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" (UID: "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:34.586375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.586258    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-data" pod "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" (UID: "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:34.586375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.586276    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.586375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.586296    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.586375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.586310    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.586375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.586325    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.587546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.587468    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-token" pod "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" (UID: "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:34.588154 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.588113    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-podinfo" pod "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" (UID: "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:06:34.588834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.588802    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-envoy" pod "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" (UID: "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:34.588960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.588913    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-kube-api-access-pkdm2" pod "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" (UID: "1b8c4e7e-20e8-42e6-92f4-4c105e41f52e"). InnerVolumeSpecName "kube-api-access-pkdm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:34.687494 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.687454    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.687494 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.687490    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.687494 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.687502    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkdm2\" (UniqueName: \"kubernetes.io/projected/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-kube-api-access-pkdm2\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.687756 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.687519    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.687756 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.687531    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.917294 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.917257    6284 generic.go:356] "Generic (PLEG): container finished" podID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerID="477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906" exitCode=0
Nov 02 14:06:34.917480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.917306    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"
Nov 02 14:06:34.917480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.917306    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" event={"ID":"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e","Type":"ContainerDied","Data":"477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906"}
Nov 02 14:06:34.917480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.917371    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92" event={"ID":"1b8c4e7e-20e8-42e6-92f4-4c105e41f52e","Type":"ContainerDied","Data":"99f4907fd6fbed5785c7b5fcce2027cfeefdc7c4e6ec1058057abc9b1b3dd919"}
Nov 02 14:06:34.917480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.917397    6284 scope.go:122] "RemoveContainer" containerID="477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906"
Nov 02 14:06:34.928964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.928921    6284 scope.go:122] "RemoveContainer" containerID="477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906"
Nov 02 14:06:34.929331 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:06:34.929311    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906\": container with ID starting with 477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906 not found: ID does not exist" containerID="477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906"
Nov 02 14:06:34.929493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.929341    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906"} err="failed to get container status \"477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906\": rpc error: code = NotFound desc = could not find container \"477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906\": container with ID starting with 477ca2b8f7da57446fcb5f04c235841ddc628cb15109ea9bb06974921c54d906 not found: ID does not exist"
Nov 02 14:06:34.980903 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.980862    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"]
Nov 02 14:06:34.984444 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:34.984406    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-8dr92"]
Nov 02 14:06:36.171394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:36.171359    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" path="/var/lib/kubelet/pods/1b8c4e7e-20e8-42e6-92f4-4c105e41f52e/volumes"
Nov 02 14:06:37.990736 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:37.990692    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"]
Nov 02 14:06:37.991610 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:37.991584    6284 cpu_manager.go:414] "containerMap: removing container" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy"
Nov 02 14:06:37.991610 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:37.991610    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy"
Nov 02 14:06:37.991820 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:37.991805    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8c4e7e-20e8-42e6-92f4-4c105e41f52e" containerName="istio-proxy"
Nov 02 14:06:37.996923 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:37.996894    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.020436 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.020387    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f66ca3-serving-cert\""
Nov 02 14:06:38.026209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.026174    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"]
Nov 02 14:06:38.031640 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.031599    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-maas-api-tls\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.031824 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.031735    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-metrics-certs\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.031900 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.031876    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7v62\" (UniqueName: \"kubernetes.io/projected/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-kube-api-access-r7v62\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.031977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.031958    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-openshift-service-ca\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.132888 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.132852    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7v62\" (UniqueName: \"kubernetes.io/projected/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-kube-api-access-r7v62\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.133087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.132911    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-openshift-service-ca\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.133087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.133061    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-maas-api-tls\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.133173 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.133107    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-metrics-certs\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.133717 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.133695    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-openshift-service-ca\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.135570 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.135552    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-maas-api-tls\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.135691 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.135667    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-metrics-certs\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.154474 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.154443    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7v62\" (UniqueName: \"kubernetes.io/projected/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-kube-api-access-r7v62\") pod \"maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.312488 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.312413    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:38.636871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.636780    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"]
Nov 02 14:06:38.640663 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:06:38.640628    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2e17b9_19b5_4e21_a105_b86a4fd457b7.slice/crio-f3ad2542babb6dc1429b25ed51986e969ba3f1db0e338512ddd69b5abb432dd6 WatchSource:0}: Error finding container f3ad2542babb6dc1429b25ed51986e969ba3f1db0e338512ddd69b5abb432dd6: Status 404 returned error can't find the container with id f3ad2542babb6dc1429b25ed51986e969ba3f1db0e338512ddd69b5abb432dd6
Nov 02 14:06:38.949429 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:38.949322    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj" event={"ID":"0c2e17b9-19b5-4e21-a105-b86a4fd457b7","Type":"ContainerStarted","Data":"f3ad2542babb6dc1429b25ed51986e969ba3f1db0e338512ddd69b5abb432dd6"}
Nov 02 14:06:39.959624 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:39.959572    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj" event={"ID":"0c2e17b9-19b5-4e21-a105-b86a4fd457b7","Type":"ContainerStarted","Data":"5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c"}
Nov 02 14:06:39.960212 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:39.959707    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:40.002449 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:40.002385    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj" podStartSLOduration=2.594256789 podStartE2EDuration="3.00236736s" totalImagesPullingTime="408.110571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:39.998482751 +0000 UTC m=+1331.957858358" watchObservedRunningTime="2026-11-02 14:06:40.00236736 +0000 UTC m=+1331.961742966"
Nov 02 14:06:43.898195 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:06:43.898150    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:06:45.973988 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:45.973921    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:06:55.298425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.298343    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"]
Nov 02 14:06:55.304277 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.304252    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.326283 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.324409    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b7489f-serving-cert\""
Nov 02 14:06:55.333149 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.332966    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"]
Nov 02 14:06:55.487647 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.487599    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f5baf-8069-4275-8582-5cd3f227e004-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.487835 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.487802    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqgcx\" (UniqueName: \"kubernetes.io/projected/1c7f5baf-8069-4275-8582-5cd3f227e004-kube-api-access-nqgcx\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.487911 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.487898    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-metrics-certs\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.487960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.487925    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.589563 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.589451    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-metrics-certs\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.589563 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.589516    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.589811 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.589594    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f5baf-8069-4275-8582-5cd3f227e004-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.589883 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.589821    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqgcx\" (UniqueName: \"kubernetes.io/projected/1c7f5baf-8069-4275-8582-5cd3f227e004-kube-api-access-nqgcx\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.590856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.590827    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f5baf-8069-4275-8582-5cd3f227e004-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.592303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.592277    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-metrics-certs\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.592604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.592580    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.611211 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.611171    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqgcx\" (UniqueName: \"kubernetes.io/projected/1c7f5baf-8069-4275-8582-5cd3f227e004-kube-api-access-nqgcx\") pod \"maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.622241 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.622208    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:55.950049 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:55.950007    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"]
Nov 02 14:06:55.952328 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:06:55.952302    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7f5baf_8069_4275_8582_5cd3f227e004.slice/crio-8cdad0a49f7d140ef384fc23a8329cb032b80b895011df0cfcfcc2dbc31a4d09 WatchSource:0}: Error finding container 8cdad0a49f7d140ef384fc23a8329cb032b80b895011df0cfcfcc2dbc31a4d09: Status 404 returned error can't find the container with id 8cdad0a49f7d140ef384fc23a8329cb032b80b895011df0cfcfcc2dbc31a4d09
Nov 02 14:06:56.094815 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:56.094769    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx" event={"ID":"1c7f5baf-8069-4275-8582-5cd3f227e004","Type":"ContainerStarted","Data":"8cdad0a49f7d140ef384fc23a8329cb032b80b895011df0cfcfcc2dbc31a4d09"}
Nov 02 14:06:57.103996 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:57.103908    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx" event={"ID":"1c7f5baf-8069-4275-8582-5cd3f227e004","Type":"ContainerStarted","Data":"b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da"}
Nov 02 14:06:57.104473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:57.104031    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:06:57.156990 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:06:57.155473    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx" podStartSLOduration=1.78559836 podStartE2EDuration="2.15545459s" totalImagesPullingTime="369.85623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:57.147835042 +0000 UTC m=+1349.107210649" watchObservedRunningTime="2026-11-02 14:06:57.15545459 +0000 UTC m=+1349.114830197"
Nov 02 14:07:03.117895 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:03.117864    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:07:09.104738 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.104698    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"]
Nov 02 14:07:09.124640 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.124603    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.137214 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.137174    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"]
Nov 02 14:07:09.215881 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.215819    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f66ca3-openshift-default-dockercfg-z7wqz\""
Nov 02 14:07:09.223990 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.223915    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-certs\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.224269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.224054    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab80603c-0dfa-4277-ada8-8d890df82ca4-istiod-ca-cert\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.224269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.224175    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-credential-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.224269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.224212    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-data\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.224269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.224240    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.224501 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.224293    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-envoy\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.224501 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.224355    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntkx\" (UniqueName: \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-kube-api-access-4ntkx\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.224501 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.224415    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-token\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.224501 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.224473    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-podinfo\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.325565 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.325516    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab80603c-0dfa-4277-ada8-8d890df82ca4-istiod-ca-cert\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.325774 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.325617    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-credential-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.325774 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.325644    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-data\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.325774 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.325670    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.325774 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.325746    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-envoy\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.326049 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.325842    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntkx\" (UniqueName: \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-kube-api-access-4ntkx\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.326049 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.326021    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-token\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.326163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.326091    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.326224 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.326186    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-credential-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.326288 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.326216    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-podinfo\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.326398 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.326379    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-certs\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.326465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.326448    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab80603c-0dfa-4277-ada8-8d890df82ca4-istiod-ca-cert\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.326771 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.326715    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-certs\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.326920 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.326789    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-data\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.328964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.328917    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-envoy\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.329073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.329004    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-podinfo\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.347511 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.347478    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-token\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.347657 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.347549    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntkx\" (UniqueName: \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-kube-api-access-4ntkx\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.479502 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.479398    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:09.822979 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.822950    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"]
Nov 02 14:07:09.825308 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:07:09.825277    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab80603c_0dfa_4277_ada8_8d890df82ca4.slice/crio-b1d6446f0065043b33fd9a8d629f9e7713fdc4d37ccabd12d61ed2c8b65d6e3f WatchSource:0}: Error finding container b1d6446f0065043b33fd9a8d629f9e7713fdc4d37ccabd12d61ed2c8b65d6e3f: Status 404 returned error can't find the container with id b1d6446f0065043b33fd9a8d629f9e7713fdc4d37ccabd12d61ed2c8b65d6e3f
Nov 02 14:07:09.828121 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.828084    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:07:09.828230 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.828166    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:07:09.828230 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:09.828199    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:07:10.228301 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:10.228212    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" event={"ID":"ab80603c-0dfa-4277-ada8-8d890df82ca4","Type":"ContainerStarted","Data":"bbdf417961606be88cdd94934109ee7c54b7ea7a07d922de66d200f615df9d27"}
Nov 02 14:07:10.228301 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:10.228272    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" event={"ID":"ab80603c-0dfa-4277-ada8-8d890df82ca4","Type":"ContainerStarted","Data":"b1d6446f0065043b33fd9a8d629f9e7713fdc4d37ccabd12d61ed2c8b65d6e3f"}
Nov 02 14:07:10.272931 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:10.272855    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" podStartSLOduration=1.272828547 podStartE2EDuration="1.272828547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:10.268007231 +0000 UTC m=+1362.227382840" watchObservedRunningTime="2026-11-02 14:07:10.272828547 +0000 UTC m=+1362.232204228"
Nov 02 14:07:10.480450 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:10.480347    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:10.482164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:10.482126    6284 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 14:07:10.482316 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:10.482199    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 14:07:11.480503 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:11.480465    6284 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 14:07:11.480957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:11.480521    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 14:07:12.480750 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:12.480712    6284 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 14:07:12.481274 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:12.480776    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 14:07:13.479868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:13.479811    6284 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 14:07:13.480089 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:13.479886    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 14:07:13.904053 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:07:13.904015    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:07:14.480506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:14.480454    6284 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 14:07:14.480740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:14.480542    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 14:07:15.480086 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:15.480045    6284 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 14:07:15.480496 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:15.480112    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 14:07:16.485607 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:16.485573    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:16.486049 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:16.485970    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:16.487018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:16.486999    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:07:25.209599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.209557    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"]
Nov 02 14:07:25.210454 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.210415    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l" podUID="d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" containerName="maas-api" containerID="cri-o://6c51c0553ea6af3e3daa9d22656120ed4d94a91a5c241a93ad0546d860ac2e61" gracePeriod=30
Nov 02 14:07:25.376466 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.376428    6284 generic.go:356] "Generic (PLEG): container finished" podID="d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" containerID="6c51c0553ea6af3e3daa9d22656120ed4d94a91a5c241a93ad0546d860ac2e61" exitCode=0
Nov 02 14:07:25.376650 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.376513    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l" event={"ID":"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6","Type":"ContainerDied","Data":"6c51c0553ea6af3e3daa9d22656120ed4d94a91a5c241a93ad0546d860ac2e61"}
Nov 02 14:07:25.612027 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.611633    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:07:25.738271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.734570    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-kube-api-access-rq4s8\" (UniqueName: \"kubernetes.io/projected/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-kube-api-access-rq4s8\") pod \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") "
Nov 02 14:07:25.738271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.734655    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-maas-api-tls\") pod \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") "
Nov 02 14:07:25.738271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.734742    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-metrics-certs\") pod \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") "
Nov 02 14:07:25.738271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.734975    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-openshift-service-ca\") pod \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\" (UID: \"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6\") "
Nov 02 14:07:25.738271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.736013    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-openshift-service-ca" pod "d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" (UID: "d0ceefcc-f51c-4e09-ac59-b3b1008e75f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:07:25.823582 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.822418    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-maas-api-tls" pod "d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" (UID: "d0ceefcc-f51c-4e09-ac59-b3b1008e75f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:07:25.835004 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.828241    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-kube-api-access-rq4s8" pod "d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" (UID: "d0ceefcc-f51c-4e09-ac59-b3b1008e75f6"). InnerVolumeSpecName "kube-api-access-rq4s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:07:25.835004 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.832602    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-metrics-certs" pod "d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" (UID: "d0ceefcc-f51c-4e09-ac59-b3b1008e75f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:07:25.838115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.837965    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:25.838115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.838021    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq4s8\" (UniqueName: \"kubernetes.io/projected/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-kube-api-access-rq4s8\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:25.838115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.838039    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:25.838115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:25.838056    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:26.385772 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:26.385743    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"
Nov 02 14:07:26.385772 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:26.385765    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l" event={"ID":"d0ceefcc-f51c-4e09-ac59-b3b1008e75f6","Type":"ContainerDied","Data":"5fafdd3f26aac45783731d72b1203890e1e9fdb3ad23d9241eb4c83c83307c23"}
Nov 02 14:07:26.386406 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:26.385831    6284 scope.go:122] "RemoveContainer" containerID="6c51c0553ea6af3e3daa9d22656120ed4d94a91a5c241a93ad0546d860ac2e61"
Nov 02 14:07:26.431429 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:26.431387    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"]
Nov 02 14:07:26.438116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:26.438028    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l"]
Nov 02 14:07:28.172211 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:28.172174    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" path="/var/lib/kubelet/pods/d0ceefcc-f51c-4e09-ac59-b3b1008e75f6/volumes"
Nov 02 14:07:28.460743 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:28.460642    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c7d2df-74b4fff887-gbr8l" podUID="d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.83:8443/health\": context deadline exceeded"
Nov 02 14:07:32.178629 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.178586    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"]
Nov 02 14:07:32.179169 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.178869    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f" podUID="fbc71d17-8210-4546-8745-60d22bc32650" containerName="maas-api" containerID="cri-o://237043630d015248be7a23a1a0ec5e98b33634a3130f3faf9281fd5c07ddf9c6" gracePeriod=30
Nov 02 14:07:32.456212 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.455922    6284 generic.go:356] "Generic (PLEG): container finished" podID="fbc71d17-8210-4546-8745-60d22bc32650" containerID="237043630d015248be7a23a1a0ec5e98b33634a3130f3faf9281fd5c07ddf9c6" exitCode=0
Nov 02 14:07:32.456212 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.456088    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f" event={"ID":"fbc71d17-8210-4546-8745-60d22bc32650","Type":"ContainerDied","Data":"237043630d015248be7a23a1a0ec5e98b33634a3130f3faf9281fd5c07ddf9c6"}
Nov 02 14:07:32.541112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.540714    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:07:32.549096 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.549057    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"]
Nov 02 14:07:32.549340 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.549282    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485" podUID="f89c53ef-367a-4e28-853e-aea9d487c9d4" containerName="payload-processing" containerID="cri-o://48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274" gracePeriod=30
Nov 02 14:07:32.610530 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.610486    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"]
Nov 02 14:07:32.611128 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.611071    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf" podUID="da0b8980-aff0-4e4d-873e-a3a0d17468b1" containerName="payload-pre-processing" containerID="cri-o://1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691" gracePeriod=30
Nov 02 14:07:32.651862 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.651811    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-maas-api-tls\") pod \"fbc71d17-8210-4546-8745-60d22bc32650\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") "
Nov 02 14:07:32.652197 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.652145    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-metrics-certs\") pod \"fbc71d17-8210-4546-8745-60d22bc32650\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") "
Nov 02 14:07:32.654067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.652747    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc71d17-8210-4546-8745-60d22bc32650-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc71d17-8210-4546-8745-60d22bc32650-openshift-service-ca\") pod \"fbc71d17-8210-4546-8745-60d22bc32650\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") "
Nov 02 14:07:32.654067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.652816    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc71d17-8210-4546-8745-60d22bc32650-kube-api-access-fjcbf\" (UniqueName: \"kubernetes.io/projected/fbc71d17-8210-4546-8745-60d22bc32650-kube-api-access-fjcbf\") pod \"fbc71d17-8210-4546-8745-60d22bc32650\" (UID: \"fbc71d17-8210-4546-8745-60d22bc32650\") "
Nov 02 14:07:32.654067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.653169    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc71d17-8210-4546-8745-60d22bc32650-openshift-service-ca" pod "fbc71d17-8210-4546-8745-60d22bc32650" (UID: "fbc71d17-8210-4546-8745-60d22bc32650"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:07:32.655234 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.655045    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc71d17-8210-4546-8745-60d22bc32650-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:32.656499 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.656474    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc71d17-8210-4546-8745-60d22bc32650-kube-api-access-fjcbf" pod "fbc71d17-8210-4546-8745-60d22bc32650" (UID: "fbc71d17-8210-4546-8745-60d22bc32650"). InnerVolumeSpecName "kube-api-access-fjcbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:07:32.664500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.664430    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-maas-api-tls" pod "fbc71d17-8210-4546-8745-60d22bc32650" (UID: "fbc71d17-8210-4546-8745-60d22bc32650"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:07:32.667351 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.667311    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-metrics-certs" pod "fbc71d17-8210-4546-8745-60d22bc32650" (UID: "fbc71d17-8210-4546-8745-60d22bc32650"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:07:32.756041 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.756005    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:32.756041 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.756039    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbc71d17-8210-4546-8745-60d22bc32650-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:32.756258 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.756052    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjcbf\" (UniqueName: \"kubernetes.io/projected/fbc71d17-8210-4546-8745-60d22bc32650-kube-api-access-fjcbf\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:32.893414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.893385    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:07:32.913483 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.913446    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:07:32.977024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.958578    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89c53ef-367a-4e28-853e-aea9d487c9d4-kube-api-access-lw9j8\" (UniqueName: \"kubernetes.io/projected/f89c53ef-367a-4e28-853e-aea9d487c9d4-kube-api-access-lw9j8\") pod \"f89c53ef-367a-4e28-853e-aea9d487c9d4\" (UID: \"f89c53ef-367a-4e28-853e-aea9d487c9d4\") "
Nov 02 14:07:32.977024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.958767    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89c53ef-367a-4e28-853e-aea9d487c9d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89c53ef-367a-4e28-853e-aea9d487c9d4-plugins-config-volume\") pod \"f89c53ef-367a-4e28-853e-aea9d487c9d4\" (UID: \"f89c53ef-367a-4e28-853e-aea9d487c9d4\") "
Nov 02 14:07:32.977024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.960168    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89c53ef-367a-4e28-853e-aea9d487c9d4-plugins-config-volume" pod "f89c53ef-367a-4e28-853e-aea9d487c9d4" (UID: "f89c53ef-367a-4e28-853e-aea9d487c9d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:07:32.977024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:32.962700    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89c53ef-367a-4e28-853e-aea9d487c9d4-kube-api-access-lw9j8" pod "f89c53ef-367a-4e28-853e-aea9d487c9d4" (UID: "f89c53ef-367a-4e28-853e-aea9d487c9d4"). InnerVolumeSpecName "kube-api-access-lw9j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:07:33.064964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.063850    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0b8980-aff0-4e4d-873e-a3a0d17468b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0b8980-aff0-4e4d-873e-a3a0d17468b1-plugins-config-volume\") pod \"da0b8980-aff0-4e4d-873e-a3a0d17468b1\" (UID: \"da0b8980-aff0-4e4d-873e-a3a0d17468b1\") "
Nov 02 14:07:33.064964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.064193    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0b8980-aff0-4e4d-873e-a3a0d17468b1-kube-api-access-kd47h\" (UniqueName: \"kubernetes.io/projected/da0b8980-aff0-4e4d-873e-a3a0d17468b1-kube-api-access-kd47h\") pod \"da0b8980-aff0-4e4d-873e-a3a0d17468b1\" (UID: \"da0b8980-aff0-4e4d-873e-a3a0d17468b1\") "
Nov 02 14:07:33.064964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.064689    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw9j8\" (UniqueName: \"kubernetes.io/projected/f89c53ef-367a-4e28-853e-aea9d487c9d4-kube-api-access-lw9j8\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:33.064964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.064710    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f89c53ef-367a-4e28-853e-aea9d487c9d4-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:33.072183 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.071670    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0b8980-aff0-4e4d-873e-a3a0d17468b1-plugins-config-volume" pod "da0b8980-aff0-4e4d-873e-a3a0d17468b1" (UID: "da0b8980-aff0-4e4d-873e-a3a0d17468b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:07:33.139994 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.138842    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0b8980-aff0-4e4d-873e-a3a0d17468b1-kube-api-access-kd47h" pod "da0b8980-aff0-4e4d-873e-a3a0d17468b1" (UID: "da0b8980-aff0-4e4d-873e-a3a0d17468b1"). InnerVolumeSpecName "kube-api-access-kd47h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:07:33.167057 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.167016    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd47h\" (UniqueName: \"kubernetes.io/projected/da0b8980-aff0-4e4d-873e-a3a0d17468b1-kube-api-access-kd47h\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:33.167057 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.167054    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0b8980-aff0-4e4d-873e-a3a0d17468b1-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:33.478180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.473235    6284 generic.go:356] "Generic (PLEG): container finished" podID="da0b8980-aff0-4e4d-873e-a3a0d17468b1" containerID="1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691" exitCode=0
Nov 02 14:07:33.478180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.473474    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"
Nov 02 14:07:33.478180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.473964    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf" event={"ID":"da0b8980-aff0-4e4d-873e-a3a0d17468b1","Type":"ContainerDied","Data":"1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691"}
Nov 02 14:07:33.478180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.474045    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf" event={"ID":"da0b8980-aff0-4e4d-873e-a3a0d17468b1","Type":"ContainerDied","Data":"862b1470296f071b088bc4b190fe795cc63ce5aa2f6f2d733c35aaeb303ccfc4"}
Nov 02 14:07:33.478180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.474077    6284 scope.go:122] "RemoveContainer" containerID="1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691"
Nov 02 14:07:33.481819 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.481788    6284 generic.go:356] "Generic (PLEG): container finished" podID="f89c53ef-367a-4e28-853e-aea9d487c9d4" containerID="48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274" exitCode=0
Nov 02 14:07:33.482054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.482038    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"
Nov 02 14:07:33.482185 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.482158    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485" event={"ID":"f89c53ef-367a-4e28-853e-aea9d487c9d4","Type":"ContainerDied","Data":"48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274"}
Nov 02 14:07:33.482243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.482224    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485" event={"ID":"f89c53ef-367a-4e28-853e-aea9d487c9d4","Type":"ContainerDied","Data":"33fff3621b1dfce07d069948e7ec9d043749a71a1a231219bc21968180306db7"}
Nov 02 14:07:33.484674 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.484650    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"
Nov 02 14:07:33.484797 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.484678    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f" event={"ID":"fbc71d17-8210-4546-8745-60d22bc32650","Type":"ContainerDied","Data":"6cc953c8aa8695e0fa18fa90afbad574dcfe12171f7d3415b8186f1b833c6195"}
Nov 02 14:07:33.496782 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.496392    6284 scope.go:122] "RemoveContainer" containerID="1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691"
Nov 02 14:07:33.496980 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:07:33.496929    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691\": container with ID starting with 1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691 not found: ID does not exist" containerID="1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691"
Nov 02 14:07:33.497086 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.496985    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691"} err="failed to get container status \"1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691\": rpc error: code = NotFound desc = could not find container \"1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691\": container with ID starting with 1e3172f0aa53352d44f0bd69b9fb4752e470e4a9a3a172766605306312910691 not found: ID does not exist"
Nov 02 14:07:33.497086 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.497013    6284 scope.go:122] "RemoveContainer" containerID="48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274"
Nov 02 14:07:33.508425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.508401    6284 scope.go:122] "RemoveContainer" containerID="48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274"
Nov 02 14:07:33.508852 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:07:33.508824    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274\": container with ID starting with 48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274 not found: ID does not exist" containerID="48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274"
Nov 02 14:07:33.509008 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.508858    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274"} err="failed to get container status \"48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274\": rpc error: code = NotFound desc = could not find container \"48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274\": container with ID starting with 48ed5a44c672cd29b46514d2158fbeb62659a421c7df5fb974751b4ac4b75274 not found: ID does not exist"
Nov 02 14:07:33.509008 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.508878    6284 scope.go:122] "RemoveContainer" containerID="237043630d015248be7a23a1a0ec5e98b33634a3130f3faf9281fd5c07ddf9c6"
Nov 02 14:07:33.560719 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.560677    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"]
Nov 02 14:07:33.576917 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.576887    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e543e0-8f76fcc9f-klpcf"]
Nov 02 14:07:33.632431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.632391    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"]
Nov 02 14:07:33.641258 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.641220    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e543e0-84cfdf6474-hc485"]
Nov 02 14:07:33.682927 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.682885    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"]
Nov 02 14:07:33.686315 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:33.686280    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e543e0-86c6564858-vr79f"]
Nov 02 14:07:34.176495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:34.176433    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0b8980-aff0-4e4d-873e-a3a0d17468b1" path="/var/lib/kubelet/pods/da0b8980-aff0-4e4d-873e-a3a0d17468b1/volumes"
Nov 02 14:07:34.177380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:34.177347    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89c53ef-367a-4e28-853e-aea9d487c9d4" path="/var/lib/kubelet/pods/f89c53ef-367a-4e28-853e-aea9d487c9d4/volumes"
Nov 02 14:07:34.178184 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:34.178158    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc71d17-8210-4546-8745-60d22bc32650" path="/var/lib/kubelet/pods/fbc71d17-8210-4546-8745-60d22bc32650/volumes"
Nov 02 14:07:43.900512 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:07:43.900469    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:07:53.161956 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.161324    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"]
Nov 02 14:07:53.161956 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.161652    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq" podUID="dc48a87e-108d-40f7-8a9e-b852edc482f1" containerName="payload-pre-processing" containerID="cri-o://9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621" gracePeriod=30
Nov 02 14:07:53.325421 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.325367    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq" podUID="dc48a87e-108d-40f7-8a9e-b852edc482f1" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.82:9004: connect: connection refused"
Nov 02 14:07:53.469607 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.469575    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:07:53.587080 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.587048    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc48a87e-108d-40f7-8a9e-b852edc482f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48a87e-108d-40f7-8a9e-b852edc482f1-plugins-config-volume\") pod \"dc48a87e-108d-40f7-8a9e-b852edc482f1\" (UID: \"dc48a87e-108d-40f7-8a9e-b852edc482f1\") "
Nov 02 14:07:53.587354 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.587335    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc48a87e-108d-40f7-8a9e-b852edc482f1-kube-api-access-btfff\" (UniqueName: \"kubernetes.io/projected/dc48a87e-108d-40f7-8a9e-b852edc482f1-kube-api-access-btfff\") pod \"dc48a87e-108d-40f7-8a9e-b852edc482f1\" (UID: \"dc48a87e-108d-40f7-8a9e-b852edc482f1\") "
Nov 02 14:07:53.587439 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.587396    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc48a87e-108d-40f7-8a9e-b852edc482f1-plugins-config-volume" pod "dc48a87e-108d-40f7-8a9e-b852edc482f1" (UID: "dc48a87e-108d-40f7-8a9e-b852edc482f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:07:53.588032 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.588006    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc48a87e-108d-40f7-8a9e-b852edc482f1-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:53.591043 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.591010    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc48a87e-108d-40f7-8a9e-b852edc482f1-kube-api-access-btfff" pod "dc48a87e-108d-40f7-8a9e-b852edc482f1" (UID: "dc48a87e-108d-40f7-8a9e-b852edc482f1"). InnerVolumeSpecName "kube-api-access-btfff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:07:53.690645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.690261    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btfff\" (UniqueName: \"kubernetes.io/projected/dc48a87e-108d-40f7-8a9e-b852edc482f1-kube-api-access-btfff\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:07:53.711092 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.707301    6284 generic.go:356] "Generic (PLEG): container finished" podID="dc48a87e-108d-40f7-8a9e-b852edc482f1" containerID="9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621" exitCode=0
Nov 02 14:07:53.711092 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.707431    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq" event={"ID":"dc48a87e-108d-40f7-8a9e-b852edc482f1","Type":"ContainerDied","Data":"9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621"}
Nov 02 14:07:53.711092 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.707481    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq" event={"ID":"dc48a87e-108d-40f7-8a9e-b852edc482f1","Type":"ContainerDied","Data":"eaf0a1ddddd5cf5fc7c8cc6fe1a815eec0a0054cda138c00d4bf47175ef89533"}
Nov 02 14:07:53.711092 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.707509    6284 scope.go:122] "RemoveContainer" containerID="9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621"
Nov 02 14:07:53.711092 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.709909    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"
Nov 02 14:07:53.752676 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.748067    6284 scope.go:122] "RemoveContainer" containerID="9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621"
Nov 02 14:07:53.752676 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:07:53.748591    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621\": container with ID starting with 9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621 not found: ID does not exist" containerID="9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621"
Nov 02 14:07:53.752676 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.748625    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621"} err="failed to get container status \"9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621\": rpc error: code = NotFound desc = could not find container \"9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621\": container with ID starting with 9e9a3e2d6f7c749715fde510e73c12fbfcd7ae78e80c3f9f6cca0af38f18e621 not found: ID does not exist"
Nov 02 14:07:53.779063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.779023    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"]
Nov 02 14:07:53.783791 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:53.783752    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ae366a-7498b75779-rv2qq"]
Nov 02 14:07:54.171265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:07:54.171231    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc48a87e-108d-40f7-8a9e-b852edc482f1" path="/var/lib/kubelet/pods/dc48a87e-108d-40f7-8a9e-b852edc482f1/volumes"
Nov 02 14:08:05.250779 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.250739    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"]
Nov 02 14:08:05.253204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.251012    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj" podUID="0c2e17b9-19b5-4e21-a105-b86a4fd457b7" containerName="maas-api" containerID="cri-o://5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c" gracePeriod=30
Nov 02 14:08:05.692344 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.692312    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:08:05.773472 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.773409    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-maas-api-tls\") pod \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") "
Nov 02 14:08:05.773758 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.773717    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-openshift-service-ca\") pod \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") "
Nov 02 14:08:05.773855 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.773777    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-metrics-certs\") pod \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") "
Nov 02 14:08:05.773945 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.773915    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-kube-api-access-r7v62\" (UniqueName: \"kubernetes.io/projected/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-kube-api-access-r7v62\") pod \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\" (UID: \"0c2e17b9-19b5-4e21-a105-b86a4fd457b7\") "
Nov 02 14:08:05.776686 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.774918    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-openshift-service-ca" pod "0c2e17b9-19b5-4e21-a105-b86a4fd457b7" (UID: "0c2e17b9-19b5-4e21-a105-b86a4fd457b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:05.778584 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.778534    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-kube-api-access-r7v62" pod "0c2e17b9-19b5-4e21-a105-b86a4fd457b7" (UID: "0c2e17b9-19b5-4e21-a105-b86a4fd457b7"). InnerVolumeSpecName "kube-api-access-r7v62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:05.836045 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.835993    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-maas-api-tls" pod "0c2e17b9-19b5-4e21-a105-b86a4fd457b7" (UID: "0c2e17b9-19b5-4e21-a105-b86a4fd457b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:05.837440 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.837409    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-metrics-certs" pod "0c2e17b9-19b5-4e21-a105-b86a4fd457b7" (UID: "0c2e17b9-19b5-4e21-a105-b86a4fd457b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:05.865957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.857895    6284 generic.go:356] "Generic (PLEG): container finished" podID="0c2e17b9-19b5-4e21-a105-b86a4fd457b7" containerID="5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c" exitCode=0
Nov 02 14:08:05.865957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.858128    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj" event={"ID":"0c2e17b9-19b5-4e21-a105-b86a4fd457b7","Type":"ContainerDied","Data":"5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c"}
Nov 02 14:08:05.865957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.858221    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj" event={"ID":"0c2e17b9-19b5-4e21-a105-b86a4fd457b7","Type":"ContainerDied","Data":"f3ad2542babb6dc1429b25ed51986e969ba3f1db0e338512ddd69b5abb432dd6"}
Nov 02 14:08:05.865957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.858258    6284 scope.go:122] "RemoveContainer" containerID="5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c"
Nov 02 14:08:05.865957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.858457    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"
Nov 02 14:08:05.876542 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.876291    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:05.876542 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.876326    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:05.876542 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.876344    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:05.876542 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.876357    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7v62\" (UniqueName: \"kubernetes.io/projected/0c2e17b9-19b5-4e21-a105-b86a4fd457b7-kube-api-access-r7v62\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:05.921073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.920362    6284 scope.go:122] "RemoveContainer" containerID="5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c"
Nov 02 14:08:05.921073 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:08:05.920770    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c\": container with ID starting with 5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c not found: ID does not exist" containerID="5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c"
Nov 02 14:08:05.921073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.920800    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c"} err="failed to get container status \"5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c\": rpc error: code = NotFound desc = could not find container \"5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c\": container with ID starting with 5fa937a9d51285722a1e8117f3bb67f743abcbfde50d2530884710b295794c4c not found: ID does not exist"
Nov 02 14:08:05.933190 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.933151    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"]
Nov 02 14:08:05.938965 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:05.938917    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f66ca3-5cc55d9f97-txpdj"]
Nov 02 14:08:06.171121 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:06.171040    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2e17b9-19b5-4e21-a105-b86a4fd457b7" path="/var/lib/kubelet/pods/0c2e17b9-19b5-4e21-a105-b86a4fd457b7/volumes"
Nov 02 14:08:07.748514 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:07.748469    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"]
Nov 02 14:08:07.749057 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:07.748759    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q" podUID="43bf93e3-524e-4165-ba2b-3d87d7855005" containerName="maas-api" containerID="cri-o://5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031" gracePeriod=30
Nov 02 14:08:08.140783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.140749    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:08:08.210444 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.210412    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bf93e3-524e-4165-ba2b-3d87d7855005-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43bf93e3-524e-4165-ba2b-3d87d7855005-openshift-service-ca\") pod \"43bf93e3-524e-4165-ba2b-3d87d7855005\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") "
Nov 02 14:08:08.210644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.210477    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-maas-api-tls\") pod \"43bf93e3-524e-4165-ba2b-3d87d7855005\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") "
Nov 02 14:08:08.210644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.210546    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-metrics-certs\") pod \"43bf93e3-524e-4165-ba2b-3d87d7855005\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") "
Nov 02 14:08:08.210644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.210574    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bf93e3-524e-4165-ba2b-3d87d7855005-kube-api-access-njbf4\" (UniqueName: \"kubernetes.io/projected/43bf93e3-524e-4165-ba2b-3d87d7855005-kube-api-access-njbf4\") pod \"43bf93e3-524e-4165-ba2b-3d87d7855005\" (UID: \"43bf93e3-524e-4165-ba2b-3d87d7855005\") "
Nov 02 14:08:08.210853 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.210823    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bf93e3-524e-4165-ba2b-3d87d7855005-openshift-service-ca" pod "43bf93e3-524e-4165-ba2b-3d87d7855005" (UID: "43bf93e3-524e-4165-ba2b-3d87d7855005"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:08.211211 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.211181    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43bf93e3-524e-4165-ba2b-3d87d7855005-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:08.213131 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.213102    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-maas-api-tls" pod "43bf93e3-524e-4165-ba2b-3d87d7855005" (UID: "43bf93e3-524e-4165-ba2b-3d87d7855005"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:08.213247 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.213161    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-metrics-certs" pod "43bf93e3-524e-4165-ba2b-3d87d7855005" (UID: "43bf93e3-524e-4165-ba2b-3d87d7855005"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:08.213413 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.213388    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bf93e3-524e-4165-ba2b-3d87d7855005-kube-api-access-njbf4" pod "43bf93e3-524e-4165-ba2b-3d87d7855005" (UID: "43bf93e3-524e-4165-ba2b-3d87d7855005"). InnerVolumeSpecName "kube-api-access-njbf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:08.333762 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.333577    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:08.333999 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.333976    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43bf93e3-524e-4165-ba2b-3d87d7855005-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:08.334107 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.334003    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njbf4\" (UniqueName: \"kubernetes.io/projected/43bf93e3-524e-4165-ba2b-3d87d7855005-kube-api-access-njbf4\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:08.884353 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.884314    6284 generic.go:356] "Generic (PLEG): container finished" podID="43bf93e3-524e-4165-ba2b-3d87d7855005" containerID="5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031" exitCode=0
Nov 02 14:08:08.884827 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.884375    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"
Nov 02 14:08:08.884827 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.884392    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q" event={"ID":"43bf93e3-524e-4165-ba2b-3d87d7855005","Type":"ContainerDied","Data":"5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031"}
Nov 02 14:08:08.884827 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.884451    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q" event={"ID":"43bf93e3-524e-4165-ba2b-3d87d7855005","Type":"ContainerDied","Data":"9baa580c332c4ee0e69db30f31985d77c2edcf77ec2c8e566cdf86c77bebeada"}
Nov 02 14:08:08.884827 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.884480    6284 scope.go:122] "RemoveContainer" containerID="5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031"
Nov 02 14:08:08.897022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.896999    6284 scope.go:122] "RemoveContainer" containerID="96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb"
Nov 02 14:08:08.909481 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.909456    6284 scope.go:122] "RemoveContainer" containerID="5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031"
Nov 02 14:08:08.909840 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:08:08.909813    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031\": container with ID starting with 5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031 not found: ID does not exist" containerID="5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031"
Nov 02 14:08:08.909993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.909846    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031"} err="failed to get container status \"5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031\": rpc error: code = NotFound desc = could not find container \"5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031\": container with ID starting with 5c175923e40c06adf7673f19cf01304c3a12b2f23d2072d4237b93af05623031 not found: ID does not exist"
Nov 02 14:08:08.909993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.909866    6284 scope.go:122] "RemoveContainer" containerID="96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb"
Nov 02 14:08:08.910234 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:08:08.910213    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb\": container with ID starting with 96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb not found: ID does not exist" containerID="96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb"
Nov 02 14:08:08.910298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.910247    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb"} err="failed to get container status \"96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb\": rpc error: code = NotFound desc = could not find container \"96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb\": container with ID starting with 96626775ed27a333741c1c839d8c8380f2835241c077cacc84334028b70b3dcb not found: ID does not exist"
Nov 02 14:08:08.947621 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.947578    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"]
Nov 02 14:08:08.955907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:08.955868    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fb206f-59d5f9cbd5-5c98q"]
Nov 02 14:08:09.680927 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:09.680881    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"]
Nov 02 14:08:09.681207 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:09.681172    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerName="istio-proxy" containerID="cri-o://bbdf417961606be88cdd94934109ee7c54b7ea7a07d922de66d200f615df9d27" gracePeriod=30
Nov 02 14:08:10.170252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:10.170221    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bf93e3-524e-4165-ba2b-3d87d7855005" path="/var/lib/kubelet/pods/43bf93e3-524e-4165-ba2b-3d87d7855005/volumes"
Nov 02 14:08:13.897104 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:08:13.897063    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:08:14.969983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:14.969838    6284 generic.go:356] "Generic (PLEG): container finished" podID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerID="bbdf417961606be88cdd94934109ee7c54b7ea7a07d922de66d200f615df9d27" exitCode=0
Nov 02 14:08:14.969983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:14.969916    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" event={"ID":"ab80603c-0dfa-4277-ada8-8d890df82ca4","Type":"ContainerDied","Data":"bbdf417961606be88cdd94934109ee7c54b7ea7a07d922de66d200f615df9d27"}
Nov 02 14:08:15.374438 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.374405    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:08:15.449758 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.449716    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-socket\") pod \"ab80603c-0dfa-4277-ada8-8d890df82ca4\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") "
Nov 02 14:08:15.449999 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.449767    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-token\" (UniqueName: \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-token\") pod \"ab80603c-0dfa-4277-ada8-8d890df82ca4\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") "
Nov 02 14:08:15.449999 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.449834    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-kube-api-access-4ntkx\" (UniqueName: \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-kube-api-access-4ntkx\") pod \"ab80603c-0dfa-4277-ada8-8d890df82ca4\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") "
Nov 02 14:08:15.449999 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.449883    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab80603c-0dfa-4277-ada8-8d890df82ca4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab80603c-0dfa-4277-ada8-8d890df82ca4-istiod-ca-cert\") pod \"ab80603c-0dfa-4277-ada8-8d890df82ca4\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") "
Nov 02 14:08:15.449999 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.449929    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-envoy\") pod \"ab80603c-0dfa-4277-ada8-8d890df82ca4\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") "
Nov 02 14:08:15.449999 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.449995    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-certs\") pod \"ab80603c-0dfa-4277-ada8-8d890df82ca4\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") "
Nov 02 14:08:15.450273 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.450136    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-podinfo\") pod \"ab80603c-0dfa-4277-ada8-8d890df82ca4\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") "
Nov 02 14:08:15.450273 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.450165    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-socket" pod "ab80603c-0dfa-4277-ada8-8d890df82ca4" (UID: "ab80603c-0dfa-4277-ada8-8d890df82ca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:15.450273 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.450214    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-data\") pod \"ab80603c-0dfa-4277-ada8-8d890df82ca4\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") "
Nov 02 14:08:15.450437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.450371    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-credential-socket\") pod \"ab80603c-0dfa-4277-ada8-8d890df82ca4\" (UID: \"ab80603c-0dfa-4277-ada8-8d890df82ca4\") "
Nov 02 14:08:15.450437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.450400    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab80603c-0dfa-4277-ada8-8d890df82ca4-istiod-ca-cert" pod "ab80603c-0dfa-4277-ada8-8d890df82ca4" (UID: "ab80603c-0dfa-4277-ada8-8d890df82ca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:15.450612 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.450580    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-certs" pod "ab80603c-0dfa-4277-ada8-8d890df82ca4" (UID: "ab80603c-0dfa-4277-ada8-8d890df82ca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:15.450890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.450858    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-data" pod "ab80603c-0dfa-4277-ada8-8d890df82ca4" (UID: "ab80603c-0dfa-4277-ada8-8d890df82ca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:15.451101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.451078    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-credential-socket" pod "ab80603c-0dfa-4277-ada8-8d890df82ca4" (UID: "ab80603c-0dfa-4277-ada8-8d890df82ca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:15.451187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.451112    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.451187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.451136    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab80603c-0dfa-4277-ada8-8d890df82ca4-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.451187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.451152    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.451187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.451167    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.453350 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.453248    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-envoy" pod "ab80603c-0dfa-4277-ada8-8d890df82ca4" (UID: "ab80603c-0dfa-4277-ada8-8d890df82ca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:15.453350 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.453309    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-kube-api-access-4ntkx" pod "ab80603c-0dfa-4277-ada8-8d890df82ca4" (UID: "ab80603c-0dfa-4277-ada8-8d890df82ca4"). InnerVolumeSpecName "kube-api-access-4ntkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:15.453551 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.453444    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-token" pod "ab80603c-0dfa-4277-ada8-8d890df82ca4" (UID: "ab80603c-0dfa-4277-ada8-8d890df82ca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:15.456084 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.456052    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-podinfo" pod "ab80603c-0dfa-4277-ada8-8d890df82ca4" (UID: "ab80603c-0dfa-4277-ada8-8d890df82ca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:08:15.552395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.552296    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.552395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.552333    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.552395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.552345    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab80603c-0dfa-4277-ada8-8d890df82ca4-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.552395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.552355    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.552395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.552364    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ntkx\" (UniqueName: \"kubernetes.io/projected/ab80603c-0dfa-4277-ada8-8d890df82ca4-kube-api-access-4ntkx\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.978298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.978199    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"
Nov 02 14:08:15.978298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.978186    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk" event={"ID":"ab80603c-0dfa-4277-ada8-8d890df82ca4","Type":"ContainerDied","Data":"b1d6446f0065043b33fd9a8d629f9e7713fdc4d37ccabd12d61ed2c8b65d6e3f"}
Nov 02 14:08:15.978803 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:15.978381    6284 scope.go:122] "RemoveContainer" containerID="bbdf417961606be88cdd94934109ee7c54b7ea7a07d922de66d200f615df9d27"
Nov 02 14:08:16.043973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:16.043895    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"]
Nov 02 14:08:16.047822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:16.047786    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-dfrdk"]
Nov 02 14:08:16.171043 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:16.171004    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" path="/var/lib/kubelet/pods/ab80603c-0dfa-4277-ada8-8d890df82ca4/volumes"
Nov 02 14:08:24.135718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.135618    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"]
Nov 02 14:08:24.136209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.135907    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5" podUID="75b32100-8dd4-4cbe-8aae-4d96d397fa8e" containerName="payload-processing" containerID="cri-o://c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e" gracePeriod=30
Nov 02 14:08:24.422306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.422258    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"]
Nov 02 14:08:24.422607 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.422576    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689" podUID="f3dc892f-bfcc-46da-9f5a-a004aa8b2931" containerName="maas-api" containerID="cri-o://a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988" gracePeriod=30
Nov 02 14:08:24.433584 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.433555    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:08:24.522507 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.522465    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-kube-api-access-g5849\" (UniqueName: \"kubernetes.io/projected/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-kube-api-access-g5849\") pod \"75b32100-8dd4-4cbe-8aae-4d96d397fa8e\" (UID: \"75b32100-8dd4-4cbe-8aae-4d96d397fa8e\") "
Nov 02 14:08:24.522699 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.522589    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-plugins-config-volume\") pod \"75b32100-8dd4-4cbe-8aae-4d96d397fa8e\" (UID: \"75b32100-8dd4-4cbe-8aae-4d96d397fa8e\") "
Nov 02 14:08:24.523985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.523951    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-plugins-config-volume" pod "75b32100-8dd4-4cbe-8aae-4d96d397fa8e" (UID: "75b32100-8dd4-4cbe-8aae-4d96d397fa8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:24.543557 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.543500    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-kube-api-access-g5849" pod "75b32100-8dd4-4cbe-8aae-4d96d397fa8e" (UID: "75b32100-8dd4-4cbe-8aae-4d96d397fa8e"). InnerVolumeSpecName "kube-api-access-g5849". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:24.624535 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.624496    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5849\" (UniqueName: \"kubernetes.io/projected/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-kube-api-access-g5849\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:24.624535 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.624533    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75b32100-8dd4-4cbe-8aae-4d96d397fa8e-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:24.757342 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.757308    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:08:24.826826 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.826779    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-metrics-certs\") pod \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") "
Nov 02 14:08:24.827030 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.827005    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-maas-api-tls\") pod \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") "
Nov 02 14:08:24.827087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.827057    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-openshift-service-ca\") pod \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") "
Nov 02 14:08:24.827154 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.827128    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-kube-api-access-crftl\" (UniqueName: \"kubernetes.io/projected/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-kube-api-access-crftl\") pod \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\" (UID: \"f3dc892f-bfcc-46da-9f5a-a004aa8b2931\") "
Nov 02 14:08:24.827482 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.827406    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-openshift-service-ca" pod "f3dc892f-bfcc-46da-9f5a-a004aa8b2931" (UID: "f3dc892f-bfcc-46da-9f5a-a004aa8b2931"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:24.827774 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.827752    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:24.829110 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.829084    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-metrics-certs" pod "f3dc892f-bfcc-46da-9f5a-a004aa8b2931" (UID: "f3dc892f-bfcc-46da-9f5a-a004aa8b2931"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:24.829243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.829148    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-maas-api-tls" pod "f3dc892f-bfcc-46da-9f5a-a004aa8b2931" (UID: "f3dc892f-bfcc-46da-9f5a-a004aa8b2931"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:24.829554 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.829536    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-kube-api-access-crftl" pod "f3dc892f-bfcc-46da-9f5a-a004aa8b2931" (UID: "f3dc892f-bfcc-46da-9f5a-a004aa8b2931"). InnerVolumeSpecName "kube-api-access-crftl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:24.928734 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.928688    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:24.928734 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.928725    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:24.928734 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:24.928736    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crftl\" (UniqueName: \"kubernetes.io/projected/f3dc892f-bfcc-46da-9f5a-a004aa8b2931-kube-api-access-crftl\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:25.140204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.139087    6284 generic.go:356] "Generic (PLEG): container finished" podID="f3dc892f-bfcc-46da-9f5a-a004aa8b2931" containerID="a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988" exitCode=0
Nov 02 14:08:25.140204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.139342    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689" event={"ID":"f3dc892f-bfcc-46da-9f5a-a004aa8b2931","Type":"ContainerDied","Data":"a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988"}
Nov 02 14:08:25.140204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.139405    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689" event={"ID":"f3dc892f-bfcc-46da-9f5a-a004aa8b2931","Type":"ContainerDied","Data":"d49c39d5bbe7bfe5fee832bc20014d8d45eabb7c09094a3e9795dcfa03e2e5dd"}
Nov 02 14:08:25.140204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.139435    6284 scope.go:122] "RemoveContainer" containerID="a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988"
Nov 02 14:08:25.140204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.139620    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"
Nov 02 14:08:25.164907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.144650    6284 generic.go:356] "Generic (PLEG): container finished" podID="75b32100-8dd4-4cbe-8aae-4d96d397fa8e" containerID="c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e" exitCode=0
Nov 02 14:08:25.164907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.144703    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5" event={"ID":"75b32100-8dd4-4cbe-8aae-4d96d397fa8e","Type":"ContainerDied","Data":"c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e"}
Nov 02 14:08:25.164907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.144756    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5" event={"ID":"75b32100-8dd4-4cbe-8aae-4d96d397fa8e","Type":"ContainerDied","Data":"e9537750bb6f4e0580b6a76af549fa45ea3113154dea86011b4b65ed1083f9cc"}
Nov 02 14:08:25.164907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.144820    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"
Nov 02 14:08:25.164907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.158159    6284 scope.go:122] "RemoveContainer" containerID="a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988"
Nov 02 14:08:25.164907 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:08:25.158792    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988\": container with ID starting with a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988 not found: ID does not exist" containerID="a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988"
Nov 02 14:08:25.164907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.158824    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988"} err="failed to get container status \"a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988\": rpc error: code = NotFound desc = could not find container \"a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988\": container with ID starting with a8dadfc71092efe19ae2bf3c4c632a8b354550e2e4b517173dc8509a0c197988 not found: ID does not exist"
Nov 02 14:08:25.164907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.158843    6284 scope.go:122] "RemoveContainer" containerID="c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e"
Nov 02 14:08:25.181816 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.181769    6284 scope.go:122] "RemoveContainer" containerID="c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e"
Nov 02 14:08:25.182272 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:08:25.182235    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e\": container with ID starting with c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e not found: ID does not exist" containerID="c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e"
Nov 02 14:08:25.182362 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.182281    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e"} err="failed to get container status \"c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e\": rpc error: code = NotFound desc = could not find container \"c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e\": container with ID starting with c07b652dfdc41c209a63f7e8c40ff5a5801b78fc35fce9a0e57dcfadee11922e not found: ID does not exist"
Nov 02 14:08:25.214694 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.214663    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"]
Nov 02 14:08:25.223173 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.223129    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-60ad62-79744ffd8-rh689"]
Nov 02 14:08:25.271875 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.271829    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"]
Nov 02 14:08:25.274945 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:25.274901    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-17a7ba-665788dcb8-2msd5"]
Nov 02 14:08:26.170729 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:26.170690    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b32100-8dd4-4cbe-8aae-4d96d397fa8e" path="/var/lib/kubelet/pods/75b32100-8dd4-4cbe-8aae-4d96d397fa8e/volumes"
Nov 02 14:08:26.171209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:26.171193    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3dc892f-bfcc-46da-9f5a-a004aa8b2931" path="/var/lib/kubelet/pods/f3dc892f-bfcc-46da-9f5a-a004aa8b2931/volumes"
Nov 02 14:08:34.708973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.708916    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"]
Nov 02 14:08:34.709719 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709695    6284 cpu_manager.go:414] "containerMap: removing container" podUID="f3dc892f-bfcc-46da-9f5a-a004aa8b2931" containerName="maas-api"
Nov 02 14:08:34.709822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709720    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3dc892f-bfcc-46da-9f5a-a004aa8b2931" containerName="maas-api"
Nov 02 14:08:34.709822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709736    6284 cpu_manager.go:414] "containerMap: removing container" podUID="43bf93e3-524e-4165-ba2b-3d87d7855005" containerName="maas-api"
Nov 02 14:08:34.709822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709743    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bf93e3-524e-4165-ba2b-3d87d7855005" containerName="maas-api"
Nov 02 14:08:34.709822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709758    6284 cpu_manager.go:414] "containerMap: removing container" podUID="0c2e17b9-19b5-4e21-a105-b86a4fd457b7" containerName="maas-api"
Nov 02 14:08:34.709822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709766    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2e17b9-19b5-4e21-a105-b86a4fd457b7" containerName="maas-api"
Nov 02 14:08:34.709822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709785    6284 cpu_manager.go:414] "containerMap: removing container" podUID="dc48a87e-108d-40f7-8a9e-b852edc482f1" containerName="payload-pre-processing"
Nov 02 14:08:34.709822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709792    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc48a87e-108d-40f7-8a9e-b852edc482f1" containerName="payload-pre-processing"
Nov 02 14:08:34.709822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709807    6284 cpu_manager.go:414] "containerMap: removing container" podUID="f89c53ef-367a-4e28-853e-aea9d487c9d4" containerName="payload-processing"
Nov 02 14:08:34.709822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709814    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89c53ef-367a-4e28-853e-aea9d487c9d4" containerName="payload-processing"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709832    6284 cpu_manager.go:414] "containerMap: removing container" podUID="43bf93e3-524e-4165-ba2b-3d87d7855005" containerName="maas-api"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709839    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bf93e3-524e-4165-ba2b-3d87d7855005" containerName="maas-api"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709853    6284 cpu_manager.go:414] "containerMap: removing container" podUID="fbc71d17-8210-4546-8745-60d22bc32650" containerName="maas-api"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709860    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc71d17-8210-4546-8745-60d22bc32650" containerName="maas-api"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709871    6284 cpu_manager.go:414] "containerMap: removing container" podUID="d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" containerName="maas-api"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709878    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" containerName="maas-api"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709902    6284 cpu_manager.go:414] "containerMap: removing container" podUID="da0b8980-aff0-4e4d-873e-a3a0d17468b1" containerName="payload-pre-processing"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709909    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0b8980-aff0-4e4d-873e-a3a0d17468b1" containerName="payload-pre-processing"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709967    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerName="istio-proxy"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709976    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerName="istio-proxy"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.709993    6284 cpu_manager.go:414] "containerMap: removing container" podUID="75b32100-8dd4-4cbe-8aae-4d96d397fa8e" containerName="payload-processing"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710000    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b32100-8dd4-4cbe-8aae-4d96d397fa8e" containerName="payload-processing"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710149    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89c53ef-367a-4e28-853e-aea9d487c9d4" containerName="payload-processing"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710162    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bf93e3-524e-4165-ba2b-3d87d7855005" containerName="maas-api"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710172    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab80603c-0dfa-4277-ada8-8d890df82ca4" containerName="istio-proxy"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710185    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc71d17-8210-4546-8745-60d22bc32650" containerName="maas-api"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710196    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ceefcc-f51c-4e09-ac59-b3b1008e75f6" containerName="maas-api"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710206    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc48a87e-108d-40f7-8a9e-b852edc482f1" containerName="payload-pre-processing"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710217    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b32100-8dd4-4cbe-8aae-4d96d397fa8e" containerName="payload-processing"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710229    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0b8980-aff0-4e4d-873e-a3a0d17468b1" containerName="payload-pre-processing"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710239    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3dc892f-bfcc-46da-9f5a-a004aa8b2931" containerName="maas-api"
Nov 02 14:08:34.710284 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.710249    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2e17b9-19b5-4e21-a105-b86a4fd457b7" containerName="maas-api"
Nov 02 14:08:34.714167 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.714140    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.733097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.733067    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-13367a-serving-cert\""
Nov 02 14:08:34.748868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.748778    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"]
Nov 02 14:08:34.833670 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.833628    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-metrics-certs\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.833872 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.833681    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8hz8\" (UniqueName: \"kubernetes.io/projected/88d1ea2b-135f-4081-b93b-bf6018aa6931-kube-api-access-l8hz8\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.833872 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.833783    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d1ea2b-135f-4081-b93b-bf6018aa6931-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.833872 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.833835    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-maas-api-tls\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.935605 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.935549    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-metrics-certs\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.935605 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.935600    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8hz8\" (UniqueName: \"kubernetes.io/projected/88d1ea2b-135f-4081-b93b-bf6018aa6931-kube-api-access-l8hz8\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.935886 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.935636    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d1ea2b-135f-4081-b93b-bf6018aa6931-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.935886 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.935673    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-maas-api-tls\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.935886 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:08:34.935837    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-serving-cert: secret "maas-api-e2e-ipp-a-13367a-serving-cert" not found
Nov 02 14:08:34.936051 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:08:34.935928    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-maas-api-tls podName:88d1ea2b-135f-4081-b93b-bf6018aa6931 nodeName:}" failed. No retries permitted until 2026-11-02 14:08:35.435905443 +0000 UTC m=+1447.395281029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-maas-api-tls") pod "maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg" (UID: "88d1ea2b-135f-4081-b93b-bf6018aa6931") : secret "maas-api-e2e-ipp-a-13367a-serving-cert" not found
Nov 02 14:08:34.936428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.936407    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d1ea2b-135f-4081-b93b-bf6018aa6931-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.938232 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.938206    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-metrics-certs\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:34.957367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:34.957336    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8hz8\" (UniqueName: \"kubernetes.io/projected/88d1ea2b-135f-4081-b93b-bf6018aa6931-kube-api-access-l8hz8\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:35.442397 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:35.442348    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-maas-api-tls\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:35.444859 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:35.444835    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-maas-api-tls\") pod \"maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:35.631236 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:35.631195    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:35.954508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:35.954418    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"]
Nov 02 14:08:35.960294 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:08:35.960264    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d1ea2b_135f_4081_b93b_bf6018aa6931.slice/crio-df4f768fecf84983b2fbcbc7ca581aa8bdcb3bfbdf76060d39be662c0b5b1c0b WatchSource:0}: Error finding container df4f768fecf84983b2fbcbc7ca581aa8bdcb3bfbdf76060d39be662c0b5b1c0b: Status 404 returned error can't find the container with id df4f768fecf84983b2fbcbc7ca581aa8bdcb3bfbdf76060d39be662c0b5b1c0b
Nov 02 14:08:36.274497 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:36.274461    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg" event={"ID":"88d1ea2b-135f-4081-b93b-bf6018aa6931","Type":"ContainerStarted","Data":"df4f768fecf84983b2fbcbc7ca581aa8bdcb3bfbdf76060d39be662c0b5b1c0b"}
Nov 02 14:08:37.284185 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:37.284147    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg" event={"ID":"88d1ea2b-135f-4081-b93b-bf6018aa6931","Type":"ContainerStarted","Data":"e3e8dcc17342bf127b0d6257166df38f8fab66b5813b0eaae24c56544e083c2a"}
Nov 02 14:08:37.284622 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:37.284231    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:37.328261 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:37.328199    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg" podStartSLOduration=2.992854069 podStartE2EDuration="3.328181012s" totalImagesPullingTime="335.326943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:37.326184036 +0000 UTC m=+1449.285559637" watchObservedRunningTime="2026-11-02 14:08:37.328181012 +0000 UTC m=+1449.287556618"
Nov 02 14:08:42.656631 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:42.656514    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"]
Nov 02 14:08:42.657507 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:42.657445    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx" podUID="1c7f5baf-8069-4275-8582-5cd3f227e004" containerName="maas-api" containerID="cri-o://b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da" gracePeriod=30
Nov 02 14:08:43.018139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.018106    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:08:43.161461 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.161417    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7f5baf-8069-4275-8582-5cd3f227e004-kube-api-access-nqgcx\" (UniqueName: \"kubernetes.io/projected/1c7f5baf-8069-4275-8582-5cd3f227e004-kube-api-access-nqgcx\") pod \"1c7f5baf-8069-4275-8582-5cd3f227e004\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") "
Nov 02 14:08:43.161653 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.161474    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7f5baf-8069-4275-8582-5cd3f227e004-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f5baf-8069-4275-8582-5cd3f227e004-openshift-service-ca\") pod \"1c7f5baf-8069-4275-8582-5cd3f227e004\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") "
Nov 02 14:08:43.161653 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.161516    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-maas-api-tls\") pod \"1c7f5baf-8069-4275-8582-5cd3f227e004\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") "
Nov 02 14:08:43.162334 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.161663    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-metrics-certs\") pod \"1c7f5baf-8069-4275-8582-5cd3f227e004\" (UID: \"1c7f5baf-8069-4275-8582-5cd3f227e004\") "
Nov 02 14:08:43.162334 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.161967    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7f5baf-8069-4275-8582-5cd3f227e004-openshift-service-ca" pod "1c7f5baf-8069-4275-8582-5cd3f227e004" (UID: "1c7f5baf-8069-4275-8582-5cd3f227e004"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:43.162468 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.162451    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7f5baf-8069-4275-8582-5cd3f227e004-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:43.164478 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.164440    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7f5baf-8069-4275-8582-5cd3f227e004-kube-api-access-nqgcx" pod "1c7f5baf-8069-4275-8582-5cd3f227e004" (UID: "1c7f5baf-8069-4275-8582-5cd3f227e004"). InnerVolumeSpecName "kube-api-access-nqgcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:43.164620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.164493    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-metrics-certs" pod "1c7f5baf-8069-4275-8582-5cd3f227e004" (UID: "1c7f5baf-8069-4275-8582-5cd3f227e004"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:43.164620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.164541    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-maas-api-tls" pod "1c7f5baf-8069-4275-8582-5cd3f227e004" (UID: "1c7f5baf-8069-4275-8582-5cd3f227e004"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:08:43.266859 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.264222    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:43.266859 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.264267    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqgcx\" (UniqueName: \"kubernetes.io/projected/1c7f5baf-8069-4275-8582-5cd3f227e004-kube-api-access-nqgcx\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:43.266859 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.264282    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c7f5baf-8069-4275-8582-5cd3f227e004-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:43.306010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.301598    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:08:43.348650 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.348611    6284 generic.go:356] "Generic (PLEG): container finished" podID="1c7f5baf-8069-4275-8582-5cd3f227e004" containerID="b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da" exitCode=0
Nov 02 14:08:43.348853 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.348656    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"
Nov 02 14:08:43.348853 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.348652    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx" event={"ID":"1c7f5baf-8069-4275-8582-5cd3f227e004","Type":"ContainerDied","Data":"b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da"}
Nov 02 14:08:43.348853 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.348729    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx" event={"ID":"1c7f5baf-8069-4275-8582-5cd3f227e004","Type":"ContainerDied","Data":"8cdad0a49f7d140ef384fc23a8329cb032b80b895011df0cfcfcc2dbc31a4d09"}
Nov 02 14:08:43.348853 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.348756    6284 scope.go:122] "RemoveContainer" containerID="b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da"
Nov 02 14:08:43.362194 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.362165    6284 scope.go:122] "RemoveContainer" containerID="b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da"
Nov 02 14:08:43.362601 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:08:43.362579    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da\": container with ID starting with b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da not found: ID does not exist" containerID="b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da"
Nov 02 14:08:43.362689 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.362613    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da"} err="failed to get container status \"b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da\": rpc error: code = NotFound desc = could not find container \"b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da\": container with ID starting with b9e7e1ca25d8aa7ca800f851bd266ccd86af4513b226517298b56a2d530d91da not found: ID does not exist"
Nov 02 14:08:43.430052 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.430013    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"]
Nov 02 14:08:43.440764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:43.440729    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b7489f-65d79cb44c-8pmcx"]
Nov 02 14:08:43.906022 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:08:43.905985    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:08:44.170167 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:44.170086    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7f5baf-8069-4275-8582-5cd3f227e004" path="/var/lib/kubelet/pods/1c7f5baf-8069-4275-8582-5cd3f227e004/volumes"
Nov 02 14:08:50.962092 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:50.962049    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"]
Nov 02 14:08:50.962695 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:50.962347    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy" containerID="cri-o://3d31d56bd471e552c6d8e8bf75ebcc9a71d44f5eec764c7b6b97333977a6962e" gracePeriod=30
Nov 02 14:08:56.435482 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.435381    6284 generic.go:356] "Generic (PLEG): container finished" podID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerID="3d31d56bd471e552c6d8e8bf75ebcc9a71d44f5eec764c7b6b97333977a6962e" exitCode=0
Nov 02 14:08:56.435482 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.435459    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" event={"ID":"366922ef-5653-4b0f-9f15-4cd7f74d7528","Type":"ContainerDied","Data":"3d31d56bd471e552c6d8e8bf75ebcc9a71d44f5eec764c7b6b97333977a6962e"}
Nov 02 14:08:56.677761 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.677731    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:08:56.761251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.761204    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-data\") pod \"366922ef-5653-4b0f-9f15-4cd7f74d7528\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") "
Nov 02 14:08:56.761488 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.761293    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-credential-socket\") pod \"366922ef-5653-4b0f-9f15-4cd7f74d7528\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") "
Nov 02 14:08:56.761488 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.761373    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-podinfo\") pod \"366922ef-5653-4b0f-9f15-4cd7f74d7528\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") "
Nov 02 14:08:56.761488 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.761457    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-kube-api-access-s59rr\" (UniqueName: \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-kube-api-access-s59rr\") pod \"366922ef-5653-4b0f-9f15-4cd7f74d7528\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") "
Nov 02 14:08:56.761720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.761492    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-certs\") pod \"366922ef-5653-4b0f-9f15-4cd7f74d7528\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") "
Nov 02 14:08:56.761720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.761538    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-socket\") pod \"366922ef-5653-4b0f-9f15-4cd7f74d7528\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") "
Nov 02 14:08:56.761720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.761583    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-envoy\") pod \"366922ef-5653-4b0f-9f15-4cd7f74d7528\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") "
Nov 02 14:08:56.761720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.761613    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-token\" (UniqueName: \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-token\") pod \"366922ef-5653-4b0f-9f15-4cd7f74d7528\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") "
Nov 02 14:08:56.761720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.761683    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366922ef-5653-4b0f-9f15-4cd7f74d7528-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/366922ef-5653-4b0f-9f15-4cd7f74d7528-istiod-ca-cert\") pod \"366922ef-5653-4b0f-9f15-4cd7f74d7528\" (UID: \"366922ef-5653-4b0f-9f15-4cd7f74d7528\") "
Nov 02 14:08:56.763246 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.762803    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366922ef-5653-4b0f-9f15-4cd7f74d7528-istiod-ca-cert" pod "366922ef-5653-4b0f-9f15-4cd7f74d7528" (UID: "366922ef-5653-4b0f-9f15-4cd7f74d7528"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:56.763246 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.763121    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-data" pod "366922ef-5653-4b0f-9f15-4cd7f74d7528" (UID: "366922ef-5653-4b0f-9f15-4cd7f74d7528"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:56.763486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.763460    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-credential-socket" pod "366922ef-5653-4b0f-9f15-4cd7f74d7528" (UID: "366922ef-5653-4b0f-9f15-4cd7f74d7528"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:56.764663 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.764636    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-socket" pod "366922ef-5653-4b0f-9f15-4cd7f74d7528" (UID: "366922ef-5653-4b0f-9f15-4cd7f74d7528"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:56.768741 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.767116    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-certs" pod "366922ef-5653-4b0f-9f15-4cd7f74d7528" (UID: "366922ef-5653-4b0f-9f15-4cd7f74d7528"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:56.770178 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.769535    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-podinfo" pod "366922ef-5653-4b0f-9f15-4cd7f74d7528" (UID: "366922ef-5653-4b0f-9f15-4cd7f74d7528"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:08:56.770431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.770396    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-envoy" pod "366922ef-5653-4b0f-9f15-4cd7f74d7528" (UID: "366922ef-5653-4b0f-9f15-4cd7f74d7528"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:56.770634 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.770612    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-token" pod "366922ef-5653-4b0f-9f15-4cd7f74d7528" (UID: "366922ef-5653-4b0f-9f15-4cd7f74d7528"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:56.771701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.771131    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-kube-api-access-s59rr" pod "366922ef-5653-4b0f-9f15-4cd7f74d7528" (UID: "366922ef-5653-4b0f-9f15-4cd7f74d7528"). InnerVolumeSpecName "kube-api-access-s59rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:56.864922 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.864881    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:56.864922 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.864916    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:56.864922 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.864926    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:56.865200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.864953    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s59rr\" (UniqueName: \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-kube-api-access-s59rr\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:56.865200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.864967    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:56.865200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.864975    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:56.865200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.864982    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:56.865200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.864990    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366922ef-5653-4b0f-9f15-4cd7f74d7528-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:56.865200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:56.864998    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/366922ef-5653-4b0f-9f15-4cd7f74d7528-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:08:57.445256 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:57.445215    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn" event={"ID":"366922ef-5653-4b0f-9f15-4cd7f74d7528","Type":"ContainerDied","Data":"ab7b0e11d91a76756cf12c48b348f417d7d39a522a9b331a08d35509f7bfb624"}
Nov 02 14:08:57.445706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:57.445278    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"
Nov 02 14:08:57.445706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:57.445303    6284 scope.go:122] "RemoveContainer" containerID="3d31d56bd471e552c6d8e8bf75ebcc9a71d44f5eec764c7b6b97333977a6962e"
Nov 02 14:08:57.521064 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:57.521023    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"]
Nov 02 14:08:57.547004 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:57.546968    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-7b6qn"]
Nov 02 14:08:58.171631 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:08:58.171596    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" path="/var/lib/kubelet/pods/366922ef-5653-4b0f-9f15-4cd7f74d7528/volumes"
Nov 02 14:09:06.111593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.111552    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"]
Nov 02 14:09:06.112238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.112220    6284 cpu_manager.go:414] "containerMap: removing container" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy"
Nov 02 14:09:06.112290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.112239    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy"
Nov 02 14:09:06.112290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.112272    6284 cpu_manager.go:414] "containerMap: removing container" podUID="1c7f5baf-8069-4275-8582-5cd3f227e004" containerName="maas-api"
Nov 02 14:09:06.112290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.112278    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7f5baf-8069-4275-8582-5cd3f227e004" containerName="maas-api"
Nov 02 14:09:06.112399 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.112353    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="366922ef-5653-4b0f-9f15-4cd7f74d7528" containerName="istio-proxy"
Nov 02 14:09:06.112399 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.112365    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7f5baf-8069-4275-8582-5cd3f227e004" containerName="maas-api"
Nov 02 14:09:06.112399 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.112374    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bf93e3-524e-4165-ba2b-3d87d7855005" containerName="maas-api"
Nov 02 14:09:06.115957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.115923    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.131841 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.131813    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-cc8d39-serving-cert\""
Nov 02 14:09:06.149453 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.149412    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"]
Nov 02 14:09:06.191630 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.191573    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ntnm\" (UniqueName: \"kubernetes.io/projected/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-kube-api-access-8ntnm\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.191913 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.191807    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-metrics-certs\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.191913 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.191836    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.192082 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.191922    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.293358 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.293316    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.293556 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.293411    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ntnm\" (UniqueName: \"kubernetes.io/projected/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-kube-api-access-8ntnm\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.293556 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.293534    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-metrics-certs\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.293711 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.293559    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.294327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.294304    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.296163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.296133    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.296278 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.296190    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-metrics-certs\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.315927 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.315884    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ntnm\" (UniqueName: \"kubernetes.io/projected/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-kube-api-access-8ntnm\") pod \"maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.438285 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.438187    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:06.831438 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:06.831360    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"]
Nov 02 14:09:06.834273 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:09:06.834230    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db9013b_1cca_4f27_b7f7_a2ae7199e2bd.slice/crio-2a533c49242bbf5fc5a91769754cc64f6a9415123907babb93f233f96ee676c2 WatchSource:0}: Error finding container 2a533c49242bbf5fc5a91769754cc64f6a9415123907babb93f233f96ee676c2: Status 404 returned error can't find the container with id 2a533c49242bbf5fc5a91769754cc64f6a9415123907babb93f233f96ee676c2
Nov 02 14:09:07.521665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:07.521626    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7" event={"ID":"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd","Type":"ContainerStarted","Data":"52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2"}
Nov 02 14:09:07.522290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:07.521708    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:07.522290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:07.521738    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7" event={"ID":"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd","Type":"ContainerStarted","Data":"2a533c49242bbf5fc5a91769754cc64f6a9415123907babb93f233f96ee676c2"}
Nov 02 14:09:09.872741 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.872664    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7" podStartSLOduration=3.555805402 podStartE2EDuration="3.872644461s" totalImagesPullingTime="316.839059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:07.568706359 +0000 UTC m=+1479.528081966" watchObservedRunningTime="2026-11-02 14:09:09.872644461 +0000 UTC m=+1481.832020070"
Nov 02 14:09:09.876306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.876270    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"]
Nov 02 14:09:09.881645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.881610    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:09.897603 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.897487    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-794667\"/\"openshift-service-ca.crt\""
Nov 02 14:09:09.911892 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.911850    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"]
Nov 02 14:09:09.912145 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.912114    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-794667\"/\"default-dockercfg-ch99m\""
Nov 02 14:09:09.912224 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.912128    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-794667\"/\"ipp-route-794667-kserve-self-signed-certs\""
Nov 02 14:09:09.912281 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.912241    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-794667\"/\"kube-root-ca.crt\""
Nov 02 14:09:09.975255 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.975212    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qtz\" (UniqueName: \"kubernetes.io/projected/4eb70645-1999-43ff-8177-e29cf03352b3-kube-api-access-45qtz\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:09.975451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.975365    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-model-cache\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:09.975451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.975418    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-home\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:09.975548 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.975525    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb70645-1999-43ff-8177-e29cf03352b3-tls-certs\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:09.975706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.975680    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-tmp-dir\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:09.975803 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:09.975754    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-dshm\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.076572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.076535    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-tmp-dir\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.076793 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.076596    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-dshm\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.076793 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.076683    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qtz\" (UniqueName: \"kubernetes.io/projected/4eb70645-1999-43ff-8177-e29cf03352b3-kube-api-access-45qtz\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.076793 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.076755    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-model-cache\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.076793 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.076788    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-home\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.077087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.076839    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb70645-1999-43ff-8177-e29cf03352b3-tls-certs\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.077087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.077029    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-tmp-dir\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.077283 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.077247    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-model-cache\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.077360 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.077316    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-home\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.079499 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.079471    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb70645-1999-43ff-8177-e29cf03352b3-tls-certs\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.079611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.079503    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-dshm\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.098551 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.098509    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qtz\" (UniqueName: \"kubernetes.io/projected/4eb70645-1999-43ff-8177-e29cf03352b3-kube-api-access-45qtz\") pod \"ipp-route-794667-kserve-68689775d5-6bj4j\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") " pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.201292 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.201197    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:10.538523 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:10.538488    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"]
Nov 02 14:09:10.542433 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:09:10.542394    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4eb70645_1999_43ff_8177_e29cf03352b3.slice/crio-222fdc5d5981069c7bb60f185fd2cb8a80bc85fa3a5f66a20f4247e37e72f598 WatchSource:0}: Error finding container 222fdc5d5981069c7bb60f185fd2cb8a80bc85fa3a5f66a20f4247e37e72f598: Status 404 returned error can't find the container with id 222fdc5d5981069c7bb60f185fd2cb8a80bc85fa3a5f66a20f4247e37e72f598
Nov 02 14:09:11.554257 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:11.554222    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j" event={"ID":"4eb70645-1999-43ff-8177-e29cf03352b3","Type":"ContainerStarted","Data":"1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407"}
Nov 02 14:09:11.554775 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:11.554266    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j" event={"ID":"4eb70645-1999-43ff-8177-e29cf03352b3","Type":"ContainerStarted","Data":"222fdc5d5981069c7bb60f185fd2cb8a80bc85fa3a5f66a20f4247e37e72f598"}
Nov 02 14:09:11.594016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:11.593930    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j" podStartSLOduration=2.593910375 podStartE2EDuration="2.593910375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:11.592084777 +0000 UTC m=+1483.551460384" watchObservedRunningTime="2026-11-02 14:09:11.593910375 +0000 UTC m=+1483.553285984"
Nov 02 14:09:12.201726 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:12.201684    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:13.535130 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:13.535099    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:09:13.901047 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:09:13.900900    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:09:20.201640 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:20.201605    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:20.217125 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:20.217094    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:20.657159 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:20.657123    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:09:24.364517 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.364474    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"]
Nov 02 14:09:24.369076 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.369050    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.387620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.387583    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.387956 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.387914    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2454243-a80d-4b6d-9602-85036725c347-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.388174 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.388159    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-metrics-certs\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.388436 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.388421    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf292\" (UniqueName: \"kubernetes.io/projected/e2454243-a80d-4b6d-9602-85036725c347-kube-api-access-hf292\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.394065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.394030    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-8acd34-serving-cert\""
Nov 02 14:09:24.414893 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.414846    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"]
Nov 02 14:09:24.489381 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.489342    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.489381 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.489384    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2454243-a80d-4b6d-9602-85036725c347-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.489617 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.489413    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-metrics-certs\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.489617 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.489496    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf292\" (UniqueName: \"kubernetes.io/projected/e2454243-a80d-4b6d-9602-85036725c347-kube-api-access-hf292\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.489617 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:09:24.489535    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-serving-cert: secret "maas-api-e2e-praxis-a-8acd34-serving-cert" not found
Nov 02 14:09:24.489617 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:09:24.489614    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-maas-api-tls podName:e2454243-a80d-4b6d-9602-85036725c347 nodeName:}" failed. No retries permitted until 2026-11-02 14:09:24.989592142 +0000 UTC m=+1496.948967727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-maas-api-tls") pod "maas-api-e2e-praxis-a-8acd34-844457576f-csr5r" (UID: "e2454243-a80d-4b6d-9602-85036725c347") : secret "maas-api-e2e-praxis-a-8acd34-serving-cert" not found
Nov 02 14:09:24.490168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.490145    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2454243-a80d-4b6d-9602-85036725c347-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.492090 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.492068    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-metrics-certs\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.512233 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.512199    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf292\" (UniqueName: \"kubernetes.io/projected/e2454243-a80d-4b6d-9602-85036725c347-kube-api-access-hf292\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.996432 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.996389    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:24.999062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:24.999034    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8acd34-844457576f-csr5r\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:25.297892 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:25.297783    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:25.632431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:25.632392    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"]
Nov 02 14:09:25.687493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:25.687447    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r" event={"ID":"e2454243-a80d-4b6d-9602-85036725c347","Type":"ContainerStarted","Data":"aff95199aa348e2901d996e9aef9f2edd94ac8920daa5ed292f2a522c2b31012"}
Nov 02 14:09:27.702220 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:27.702116    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r" event={"ID":"e2454243-a80d-4b6d-9602-85036725c347","Type":"ContainerStarted","Data":"48df3d41908aa966b3fcb13b141e4036eef092fc0c4854e832696144ff65a434"}
Nov 02 14:09:27.702716 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:27.702327    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:27.743620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:27.743552    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r" podStartSLOduration=2.085485097 podStartE2EDuration="3.74353085s" totalImagesPullingTime="1.658045753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:27.740791532 +0000 UTC m=+1499.700167139" watchObservedRunningTime="2026-11-02 14:09:27.74353085 +0000 UTC m=+1499.702906459"
Nov 02 14:09:28.431679 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:28.431647    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:09:28.434495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:28.434466    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:09:29.326654 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:29.326623    6284 scope.go:122] "RemoveContainer" containerID="e34b9e5ccadcbe031779c13eee4aa81233dbf4f2c7449e15aea1d97c9e93ed19"
Nov 02 14:09:33.711037 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:33.710991    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r" podUID="e2454243-a80d-4b6d-9602-85036725c347" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.93:8443/health\": dial tcp 10.134.0.93:8443: connect: connection refused"
Nov 02 14:09:34.868261 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:34.868220    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"]
Nov 02 14:09:34.873058 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:34.873028    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:34.889195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:34.889165    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c1d69c-serving-cert\""
Nov 02 14:09:34.893412 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:34.893377    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"]
Nov 02 14:09:34.926628 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:34.926573    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:34.926843 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:34.926637    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27xtp\" (UniqueName: \"kubernetes.io/projected/658fde51-0a39-44e5-974c-1a1deeb86b47-kube-api-access-27xtp\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:34.926915 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:34.926891    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/658fde51-0a39-44e5-974c-1a1deeb86b47-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:34.927150 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:34.927126    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-metrics-certs\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:35.028662 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.028626    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-metrics-certs\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:35.028871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.028725    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:35.028871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.028749    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xtp\" (UniqueName: \"kubernetes.io/projected/658fde51-0a39-44e5-974c-1a1deeb86b47-kube-api-access-27xtp\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:35.028871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.028833    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/658fde51-0a39-44e5-974c-1a1deeb86b47-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:35.029573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.029552    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/658fde51-0a39-44e5-974c-1a1deeb86b47-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:35.031199 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.031180    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-metrics-certs\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:35.031421 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.031399    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:35.050811 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.050776    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xtp\" (UniqueName: \"kubernetes.io/projected/658fde51-0a39-44e5-974c-1a1deeb86b47-kube-api-access-27xtp\") pod \"maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:35.187100 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.187005    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:35.506568 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.506530    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"]
Nov 02 14:09:35.509118 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:09:35.509076    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658fde51_0a39_44e5_974c_1a1deeb86b47.slice/crio-cead7e4b5299809d612d884918dbb7dcad71648f2cdbc389203abddf3ce3c0af WatchSource:0}: Error finding container cead7e4b5299809d612d884918dbb7dcad71648f2cdbc389203abddf3ce3c0af: Status 404 returned error can't find the container with id cead7e4b5299809d612d884918dbb7dcad71648f2cdbc389203abddf3ce3c0af
Nov 02 14:09:35.761863 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:35.761754    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86" event={"ID":"658fde51-0a39-44e5-974c-1a1deeb86b47","Type":"ContainerStarted","Data":"cead7e4b5299809d612d884918dbb7dcad71648f2cdbc389203abddf3ce3c0af"}
Nov 02 14:09:36.769588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:36.769551    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86" event={"ID":"658fde51-0a39-44e5-974c-1a1deeb86b47","Type":"ContainerStarted","Data":"d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf"}
Nov 02 14:09:36.770088 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:36.769746    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:36.808672 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:36.808613    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86" podStartSLOduration=2.432404556 podStartE2EDuration="2.808596964s" totalImagesPullingTime="376.192408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:36.806169674 +0000 UTC m=+1508.765545281" watchObservedRunningTime="2026-11-02 14:09:36.808596964 +0000 UTC m=+1508.767972570"
Nov 02 14:09:37.778639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:37.778604    6284 generic.go:356] "Generic (PLEG): container finished" podID="e2454243-a80d-4b6d-9602-85036725c347" containerID="48df3d41908aa966b3fcb13b141e4036eef092fc0c4854e832696144ff65a434" exitCode=1
Nov 02 14:09:37.779090 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:37.778675    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r" event={"ID":"e2454243-a80d-4b6d-9602-85036725c347","Type":"ContainerDied","Data":"48df3d41908aa966b3fcb13b141e4036eef092fc0c4854e832696144ff65a434"}
Nov 02 14:09:37.779168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:37.779155    6284 scope.go:122] "RemoveContainer" containerID="48df3d41908aa966b3fcb13b141e4036eef092fc0c4854e832696144ff65a434"
Nov 02 14:09:38.789826 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:38.789790    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r" event={"ID":"e2454243-a80d-4b6d-9602-85036725c347","Type":"ContainerStarted","Data":"6c125922fde4fa6cd1f6d5209a93ac967205a5d51879b92893e43d319b7470ec"}
Nov 02 14:09:38.790379 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:38.790166    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:09:42.787062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:42.787028    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:09:43.904772 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:09:43.904732    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:09:44.821420 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:09:44.821389    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:10:06.574994 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.574954    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"]
Nov 02 14:10:06.579991 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.579920    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.596123 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.596087    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-370905-serving-cert\""
Nov 02 14:10:06.610527 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.610132    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"]
Nov 02 14:10:06.739243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.739203    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd924467-a73e-4ce4-b1ba-7843d4496790-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.739473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.739269    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-metrics-certs\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.739473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.739446    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-maas-api-tls\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.739604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.739546    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdjs\" (UniqueName: \"kubernetes.io/projected/bd924467-a73e-4ce4-b1ba-7843d4496790-kube-api-access-wzdjs\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.841035 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.840925    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd924467-a73e-4ce4-b1ba-7843d4496790-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.841035 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.841012    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-metrics-certs\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.841306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.841092    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-maas-api-tls\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.841306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.841156    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdjs\" (UniqueName: \"kubernetes.io/projected/bd924467-a73e-4ce4-b1ba-7843d4496790-kube-api-access-wzdjs\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.841735 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.841707    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd924467-a73e-4ce4-b1ba-7843d4496790-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.843719 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.843688    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-metrics-certs\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.843884 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.843860    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-maas-api-tls\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.862975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.862918    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdjs\" (UniqueName: \"kubernetes.io/projected/bd924467-a73e-4ce4-b1ba-7843d4496790-kube-api-access-wzdjs\") pod \"maas-api-e2e-praxis-b-370905-d6547898c-6mcgn\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:06.911549 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:06.911512    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:07.241758 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:07.241724    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"]
Nov 02 14:10:07.244450 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:10:07.244412    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd924467_a73e_4ce4_b1ba_7843d4496790.slice/crio-700f918bbacb7b1bce8eff2f50d8613dd7f552733d70a67bdf5369a9839c5469 WatchSource:0}: Error finding container 700f918bbacb7b1bce8eff2f50d8613dd7f552733d70a67bdf5369a9839c5469: Status 404 returned error can't find the container with id 700f918bbacb7b1bce8eff2f50d8613dd7f552733d70a67bdf5369a9839c5469
Nov 02 14:10:08.043010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:08.042977    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn" event={"ID":"bd924467-a73e-4ce4-b1ba-7843d4496790","Type":"ContainerStarted","Data":"5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd"}
Nov 02 14:10:08.043468 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:08.043041    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn" event={"ID":"bd924467-a73e-4ce4-b1ba-7843d4496790","Type":"ContainerStarted","Data":"700f918bbacb7b1bce8eff2f50d8613dd7f552733d70a67bdf5369a9839c5469"}
Nov 02 14:10:08.043468 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:08.043173    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:08.116967 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:08.116894    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn" podStartSLOduration=1.7878106059999999 podStartE2EDuration="2.116874894s" totalImagesPullingTime="329.064288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:08.113865814 +0000 UTC m=+1540.073241418" watchObservedRunningTime="2026-11-02 14:10:08.116874894 +0000 UTC m=+1540.076250501"
Nov 02 14:10:10.260523 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.260490    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"]
Nov 02 14:10:10.265229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.265202    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.280068 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.280027    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-de493d-serving-cert\""
Nov 02 14:10:10.299988 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.295880    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"]
Nov 02 14:10:10.398005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.397965    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-metrics-certs\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.398281 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.398116    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc08500-ee23-49bb-acb6-caa893c5b372-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.398281 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.398182    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-maas-api-tls\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.398281 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.398267    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2tpm\" (UniqueName: \"kubernetes.io/projected/8bc08500-ee23-49bb-acb6-caa893c5b372-kube-api-access-t2tpm\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.504503 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.502382    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-metrics-certs\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.504503 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.502490    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc08500-ee23-49bb-acb6-caa893c5b372-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.504503 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.502551    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-maas-api-tls\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.504503 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.502596    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2tpm\" (UniqueName: \"kubernetes.io/projected/8bc08500-ee23-49bb-acb6-caa893c5b372-kube-api-access-t2tpm\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.504869 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.504638    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc08500-ee23-49bb-acb6-caa893c5b372-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.506923 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.506893    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-metrics-certs\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.507690 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.507656    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-maas-api-tls\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.524460 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.524382    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2tpm\" (UniqueName: \"kubernetes.io/projected/8bc08500-ee23-49bb-acb6-caa893c5b372-kube-api-access-t2tpm\") pod \"maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.581730 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.581696    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:10.897944 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.897812    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"]
Nov 02 14:10:10.904882 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:10:10.904851    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc08500_ee23_49bb_acb6_caa893c5b372.slice/crio-b40a31bdf5c31ab5c2a2c772de4a3dcdbfa4885190804ac36654f9c6486c8cb4 WatchSource:0}: Error finding container b40a31bdf5c31ab5c2a2c772de4a3dcdbfa4885190804ac36654f9c6486c8cb4: Status 404 returned error can't find the container with id b40a31bdf5c31ab5c2a2c772de4a3dcdbfa4885190804ac36654f9c6486c8cb4
Nov 02 14:10:10.906393 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:10.906374    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:10:11.065852 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:11.065814    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89" event={"ID":"8bc08500-ee23-49bb-acb6-caa893c5b372","Type":"ContainerStarted","Data":"b40a31bdf5c31ab5c2a2c772de4a3dcdbfa4885190804ac36654f9c6486c8cb4"}
Nov 02 14:10:12.075272 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:12.075235    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89" event={"ID":"8bc08500-ee23-49bb-acb6-caa893c5b372","Type":"ContainerStarted","Data":"c2e16bc29170e5b75c1802ae13b9edc32cf13e0fe14ba4e6da3587017baaef43"}
Nov 02 14:10:12.075699 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:12.075341    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:12.116847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:12.116781    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89" podStartSLOduration=1.711308378 podStartE2EDuration="2.116764114s" totalImagesPullingTime="405.455736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:12.114702976 +0000 UTC m=+1544.074078584" watchObservedRunningTime="2026-11-02 14:10:12.116764114 +0000 UTC m=+1544.076139722"
Nov 02 14:10:13.896108 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:10:13.896066    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:10:14.056733 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:14.056701    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:10:18.087856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:18.087823    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:10:26.179561 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.179526    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"]
Nov 02 14:10:26.184085 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.184064    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.230607 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.230566    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"]
Nov 02 14:10:26.251377 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.251345    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-732d3a-serving-cert\""
Nov 02 14:10:26.337061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.337011    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.337061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.337059    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xjh\" (UniqueName: \"kubernetes.io/projected/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-kube-api-access-f4xjh\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.337292 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.337085    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.337292 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.337221    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-metrics-certs\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.438703 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.438597    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.438703 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.438653    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xjh\" (UniqueName: \"kubernetes.io/projected/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-kube-api-access-f4xjh\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.438703 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.438681    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.438703 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.438704    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-metrics-certs\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.439581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.439549    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.441930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.441887    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-metrics-certs\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.442122 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.442021    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.464704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.462997    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xjh\" (UniqueName: \"kubernetes.io/projected/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-kube-api-access-f4xjh\") pod \"maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.499885 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.499852    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:26.825336 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:26.825305    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"]
Nov 02 14:10:26.827214 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:10:26.827188    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c9b9a4_16a1_434e_8e6c_4eaa74c599e9.slice/crio-bc19a18a0291c75ca0d38d2d4aae559595d72b30482ce89cdef3e8550fa3a40e WatchSource:0}: Error finding container bc19a18a0291c75ca0d38d2d4aae559595d72b30482ce89cdef3e8550fa3a40e: Status 404 returned error can't find the container with id bc19a18a0291c75ca0d38d2d4aae559595d72b30482ce89cdef3e8550fa3a40e
Nov 02 14:10:27.224456 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:27.224409    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj" event={"ID":"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9","Type":"ContainerStarted","Data":"bc19a18a0291c75ca0d38d2d4aae559595d72b30482ce89cdef3e8550fa3a40e"}
Nov 02 14:10:28.232992 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:28.232958    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj" event={"ID":"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9","Type":"ContainerStarted","Data":"ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a"}
Nov 02 14:10:28.277533 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:28.277468    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj" podStartSLOduration=1.955622878 podStartE2EDuration="2.277442409s" totalImagesPullingTime="321.819531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:28.272974483 +0000 UTC m=+1560.232350091" watchObservedRunningTime="2026-11-02 14:10:28.277442409 +0000 UTC m=+1560.236818017"
Nov 02 14:10:29.241012 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:29.240976    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:30.273618 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:30.273579    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"]
Nov 02 14:10:30.274074 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:30.273858    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy" containerID="cri-o://9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8" gracePeriod=30
Nov 02 14:10:33.597235 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:33.597177    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"]
Nov 02 14:10:33.597786 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:33.597555    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy" containerID="cri-o://f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc" gracePeriod=30
Nov 02 14:10:35.254758 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.254721    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:10:35.573785 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.573754    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 14:10:35.668034 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.667972    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/112fb288-71fc-4554-a30d-2f7776fb51ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112fb288-71fc-4554-a30d-2f7776fb51ea-istiod-ca-cert\") pod \"112fb288-71fc-4554-a30d-2f7776fb51ea\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") "
Nov 02 14:10:35.668248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668063    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-token\" (UniqueName: \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-token\") pod \"112fb288-71fc-4554-a30d-2f7776fb51ea\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") "
Nov 02 14:10:35.668248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668146    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-credential-socket\") pod \"112fb288-71fc-4554-a30d-2f7776fb51ea\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") "
Nov 02 14:10:35.668248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668174    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-kube-api-access-bhhm4\" (UniqueName: \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-kube-api-access-bhhm4\") pod \"112fb288-71fc-4554-a30d-2f7776fb51ea\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") "
Nov 02 14:10:35.668437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668277    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-data\") pod \"112fb288-71fc-4554-a30d-2f7776fb51ea\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") "
Nov 02 14:10:35.668437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668348    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-certs\") pod \"112fb288-71fc-4554-a30d-2f7776fb51ea\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") "
Nov 02 14:10:35.668437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668391    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-envoy\") pod \"112fb288-71fc-4554-a30d-2f7776fb51ea\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") "
Nov 02 14:10:35.668437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668402    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112fb288-71fc-4554-a30d-2f7776fb51ea-istiod-ca-cert" pod "112fb288-71fc-4554-a30d-2f7776fb51ea" (UID: "112fb288-71fc-4554-a30d-2f7776fb51ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:35.668647 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668551    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-podinfo\") pod \"112fb288-71fc-4554-a30d-2f7776fb51ea\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") "
Nov 02 14:10:35.668647 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668586    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-socket\") pod \"112fb288-71fc-4554-a30d-2f7776fb51ea\" (UID: \"112fb288-71fc-4554-a30d-2f7776fb51ea\") "
Nov 02 14:10:35.668755 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668675    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-data" pod "112fb288-71fc-4554-a30d-2f7776fb51ea" (UID: "112fb288-71fc-4554-a30d-2f7776fb51ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:35.668755 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668723    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-certs" pod "112fb288-71fc-4554-a30d-2f7776fb51ea" (UID: "112fb288-71fc-4554-a30d-2f7776fb51ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:35.668984 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.668960    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-socket" pod "112fb288-71fc-4554-a30d-2f7776fb51ea" (UID: "112fb288-71fc-4554-a30d-2f7776fb51ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:35.669199 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.669174    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-credential-socket" pod "112fb288-71fc-4554-a30d-2f7776fb51ea" (UID: "112fb288-71fc-4554-a30d-2f7776fb51ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:35.669323 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.669305    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:35.669394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.669328    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:35.669394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.669345    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:35.669394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.669359    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112fb288-71fc-4554-a30d-2f7776fb51ea-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:35.669394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.669373    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:35.671107 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.671082    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-podinfo" pod "112fb288-71fc-4554-a30d-2f7776fb51ea" (UID: "112fb288-71fc-4554-a30d-2f7776fb51ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:10:35.671667 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.671619    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-envoy" pod "112fb288-71fc-4554-a30d-2f7776fb51ea" (UID: "112fb288-71fc-4554-a30d-2f7776fb51ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:35.672048 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.672014    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-kube-api-access-bhhm4" pod "112fb288-71fc-4554-a30d-2f7776fb51ea" (UID: "112fb288-71fc-4554-a30d-2f7776fb51ea"). InnerVolumeSpecName "kube-api-access-bhhm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:35.680985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.680927    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-token" pod "112fb288-71fc-4554-a30d-2f7776fb51ea" (UID: "112fb288-71fc-4554-a30d-2f7776fb51ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:35.770226 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.770188    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:35.770226 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.770222    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:35.770226 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.770231    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhhm4\" (UniqueName: \"kubernetes.io/projected/112fb288-71fc-4554-a30d-2f7776fb51ea-kube-api-access-bhhm4\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:35.770226 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:35.770241    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112fb288-71fc-4554-a30d-2f7776fb51ea-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:36.291362 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:36.291324    6284 generic.go:356] "Generic (PLEG): container finished" podID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerID="9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8" exitCode=0
Nov 02 14:10:36.291896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:36.291417    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"
Nov 02 14:10:36.291896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:36.291438    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" event={"ID":"112fb288-71fc-4554-a30d-2f7776fb51ea","Type":"ContainerDied","Data":"9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8"}
Nov 02 14:10:36.291896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:36.291528    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4" event={"ID":"112fb288-71fc-4554-a30d-2f7776fb51ea","Type":"ContainerDied","Data":"bfaafa95286166a49f214cd160f81abd4ea8d0441215a154d3debbf4c9ad190c"}
Nov 02 14:10:36.291896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:36.291572    6284 scope.go:122] "RemoveContainer" containerID="9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8"
Nov 02 14:10:36.303355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:36.303324    6284 scope.go:122] "RemoveContainer" containerID="9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8"
Nov 02 14:10:36.304601 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:10:36.304527    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8\": container with ID starting with 9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8 not found: ID does not exist" containerID="9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8"
Nov 02 14:10:36.304601 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:36.304575    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8"} err="failed to get container status \"9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8\": rpc error: code = NotFound desc = could not find container \"9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8\": container with ID starting with 9d452c1a8f666e6cf929e33cdb62fc13a37505e8a8b614c0d6a7fd01d74949d8 not found: ID does not exist"
Nov 02 14:10:36.346694 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:36.346656    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"]
Nov 02 14:10:36.353877 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:36.353852    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-kkgp4"]
Nov 02 14:10:38.179042 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.179000    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" path="/var/lib/kubelet/pods/112fb288-71fc-4554-a30d-2f7776fb51ea/volumes"
Nov 02 14:10:38.870390 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.870359    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 14:10:38.908829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.908731    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-kube-api-access-9n7s2\" (UniqueName: \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-kube-api-access-9n7s2\") pod \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") "
Nov 02 14:10:38.908829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.908806    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-token\" (UniqueName: \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-token\") pod \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") "
Nov 02 14:10:38.909130 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.908866    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-socket\") pod \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") "
Nov 02 14:10:38.909130 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909005    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-podinfo\") pod \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") "
Nov 02 14:10:38.909130 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909077    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-certs\") pod \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") "
Nov 02 14:10:38.909305 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909229    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-socket" pod "323cdaf7-0099-49fe-ac81-d64d7acd09c0" (UID: "323cdaf7-0099-49fe-ac81-d64d7acd09c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:38.909381 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909318    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-credential-socket\") pod \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") "
Nov 02 14:10:38.909381 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909337    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-certs" pod "323cdaf7-0099-49fe-ac81-d64d7acd09c0" (UID: "323cdaf7-0099-49fe-ac81-d64d7acd09c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:38.909381 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909373    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-data\") pod \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") "
Nov 02 14:10:38.909547 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909422    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istiod-ca-cert\") pod \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") "
Nov 02 14:10:38.909547 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909500    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-envoy\") pod \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\" (UID: \"323cdaf7-0099-49fe-ac81-d64d7acd09c0\") "
Nov 02 14:10:38.909656 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909566    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-credential-socket" pod "323cdaf7-0099-49fe-ac81-d64d7acd09c0" (UID: "323cdaf7-0099-49fe-ac81-d64d7acd09c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:38.909656 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909614    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-data" pod "323cdaf7-0099-49fe-ac81-d64d7acd09c0" (UID: "323cdaf7-0099-49fe-ac81-d64d7acd09c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:38.909871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.909834    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istiod-ca-cert" pod "323cdaf7-0099-49fe-ac81-d64d7acd09c0" (UID: "323cdaf7-0099-49fe-ac81-d64d7acd09c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:38.910340 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.910314    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:38.910340 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.910340    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:38.910500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.910356    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:38.910500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.910373    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:38.910500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.910388    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:38.911308 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.911280    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-kube-api-access-9n7s2" pod "323cdaf7-0099-49fe-ac81-d64d7acd09c0" (UID: "323cdaf7-0099-49fe-ac81-d64d7acd09c0"). InnerVolumeSpecName "kube-api-access-9n7s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:38.911625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.911605    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-token" pod "323cdaf7-0099-49fe-ac81-d64d7acd09c0" (UID: "323cdaf7-0099-49fe-ac81-d64d7acd09c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:38.911722 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.911656    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-podinfo" pod "323cdaf7-0099-49fe-ac81-d64d7acd09c0" (UID: "323cdaf7-0099-49fe-ac81-d64d7acd09c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:10:38.911849 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:38.911832    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-envoy" pod "323cdaf7-0099-49fe-ac81-d64d7acd09c0" (UID: "323cdaf7-0099-49fe-ac81-d64d7acd09c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:39.011783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.011746    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n7s2\" (UniqueName: \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-kube-api-access-9n7s2\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:39.011783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.011785    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:39.012051 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.011801    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:39.012051 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.011816    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323cdaf7-0099-49fe-ac81-d64d7acd09c0-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:39.314675 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.314642    6284 generic.go:356] "Generic (PLEG): container finished" podID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerID="f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc" exitCode=0
Nov 02 14:10:39.315178 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.314706    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"
Nov 02 14:10:39.315178 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.314730    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" event={"ID":"323cdaf7-0099-49fe-ac81-d64d7acd09c0","Type":"ContainerDied","Data":"f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc"}
Nov 02 14:10:39.315178 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.314791    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w" event={"ID":"323cdaf7-0099-49fe-ac81-d64d7acd09c0","Type":"ContainerDied","Data":"5e4361fdfc60f4e0065edade1b0af5344f68b4318926e40e2c393e79f576c951"}
Nov 02 14:10:39.315178 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.314813    6284 scope.go:122] "RemoveContainer" containerID="f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc"
Nov 02 14:10:39.327206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.327179    6284 scope.go:122] "RemoveContainer" containerID="f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc"
Nov 02 14:10:39.327544 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:10:39.327527    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc\": container with ID starting with f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc not found: ID does not exist" containerID="f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc"
Nov 02 14:10:39.327591 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.327556    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc"} err="failed to get container status \"f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc\": rpc error: code = NotFound desc = could not find container \"f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc\": container with ID starting with f7f9f73adbec5960f639247c66393f02cf19b3d47106ce29a42d6520e26162dc not found: ID does not exist"
Nov 02 14:10:39.366796 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.366722    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"]
Nov 02 14:10:39.375176 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:39.375139    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hwr8w"]
Nov 02 14:10:40.172210 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:40.172174    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" path="/var/lib/kubelet/pods/323cdaf7-0099-49fe-ac81-d64d7acd09c0/volumes"
Nov 02 14:10:43.897325 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:10:43.897286    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:10:58.106725 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.106687    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"]
Nov 02 14:10:58.107312 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.107008    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86" podUID="658fde51-0a39-44e5-974c-1a1deeb86b47" containerName="maas-api" containerID="cri-o://d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf" gracePeriod=30
Nov 02 14:10:58.463596 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.463564    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:10:58.511649 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.511600    6284 generic.go:356] "Generic (PLEG): container finished" podID="658fde51-0a39-44e5-974c-1a1deeb86b47" containerID="d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf" exitCode=0
Nov 02 14:10:58.511848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.511643    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86" event={"ID":"658fde51-0a39-44e5-974c-1a1deeb86b47","Type":"ContainerDied","Data":"d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf"}
Nov 02 14:10:58.511848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.511720    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86" event={"ID":"658fde51-0a39-44e5-974c-1a1deeb86b47","Type":"ContainerDied","Data":"cead7e4b5299809d612d884918dbb7dcad71648f2cdbc389203abddf3ce3c0af"}
Nov 02 14:10:58.511848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.511656    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"
Nov 02 14:10:58.511848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.511757    6284 scope.go:122] "RemoveContainer" containerID="d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf"
Nov 02 14:10:58.524883 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.524856    6284 scope.go:122] "RemoveContainer" containerID="d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf"
Nov 02 14:10:58.525332 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:10:58.525304    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf\": container with ID starting with d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf not found: ID does not exist" containerID="d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf"
Nov 02 14:10:58.525423 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.525348    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf"} err="failed to get container status \"d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf\": rpc error: code = NotFound desc = could not find container \"d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf\": container with ID starting with d7dfefb22eb43b5f7abc9e98431cd303fab8be518465ef72d79b50679a567cdf not found: ID does not exist"
Nov 02 14:10:58.592326 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.592288    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/658fde51-0a39-44e5-974c-1a1deeb86b47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/658fde51-0a39-44e5-974c-1a1deeb86b47-openshift-service-ca\") pod \"658fde51-0a39-44e5-974c-1a1deeb86b47\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") "
Nov 02 14:10:58.592539 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.592519    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-maas-api-tls\") pod \"658fde51-0a39-44e5-974c-1a1deeb86b47\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") "
Nov 02 14:10:58.592601 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.592577    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-metrics-certs\") pod \"658fde51-0a39-44e5-974c-1a1deeb86b47\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") "
Nov 02 14:10:58.592643 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.592614    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658fde51-0a39-44e5-974c-1a1deeb86b47-kube-api-access-27xtp\" (UniqueName: \"kubernetes.io/projected/658fde51-0a39-44e5-974c-1a1deeb86b47-kube-api-access-27xtp\") pod \"658fde51-0a39-44e5-974c-1a1deeb86b47\" (UID: \"658fde51-0a39-44e5-974c-1a1deeb86b47\") "
Nov 02 14:10:58.592706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.592677    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658fde51-0a39-44e5-974c-1a1deeb86b47-openshift-service-ca" pod "658fde51-0a39-44e5-974c-1a1deeb86b47" (UID: "658fde51-0a39-44e5-974c-1a1deeb86b47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:58.593278 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.593252    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/658fde51-0a39-44e5-974c-1a1deeb86b47-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:58.594909 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.594876    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-metrics-certs" pod "658fde51-0a39-44e5-974c-1a1deeb86b47" (UID: "658fde51-0a39-44e5-974c-1a1deeb86b47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:10:58.595213 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.595193    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-maas-api-tls" pod "658fde51-0a39-44e5-974c-1a1deeb86b47" (UID: "658fde51-0a39-44e5-974c-1a1deeb86b47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:10:58.595304 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.595284    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658fde51-0a39-44e5-974c-1a1deeb86b47-kube-api-access-27xtp" pod "658fde51-0a39-44e5-974c-1a1deeb86b47" (UID: "658fde51-0a39-44e5-974c-1a1deeb86b47"). InnerVolumeSpecName "kube-api-access-27xtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:58.706026 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.705080    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:58.706026 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.705120    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/658fde51-0a39-44e5-974c-1a1deeb86b47-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:58.706026 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.705135    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27xtp\" (UniqueName: \"kubernetes.io/projected/658fde51-0a39-44e5-974c-1a1deeb86b47-kube-api-access-27xtp\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:10:58.853792 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.853745    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"]
Nov 02 14:10:58.863811 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:10:58.863770    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c1d69c-6d79b46dc9-kxv86"]
Nov 02 14:11:00.176187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:00.176153    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="658fde51-0a39-44e5-974c-1a1deeb86b47" path="/var/lib/kubelet/pods/658fde51-0a39-44e5-974c-1a1deeb86b47/volumes"
Nov 02 14:11:03.347222 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:03.347177    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"]
Nov 02 14:11:13.895325 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:11:13.895283    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:11:15.080426 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:15.080388    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"]
Nov 02 14:11:15.080990 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:15.080665    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy" containerID="cri-o://965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf" gracePeriod=30
Nov 02 14:11:18.369993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.369906    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j" podUID="4eb70645-1999-43ff-8177-e29cf03352b3" containerName="main" containerID="cri-o://1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407" gracePeriod=45
Nov 02 14:11:18.659021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.658991    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:11:18.709880 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.709843    6284 generic.go:356] "Generic (PLEG): container finished" podID="4eb70645-1999-43ff-8177-e29cf03352b3" containerID="1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407" exitCode=0
Nov 02 14:11:18.710223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.709913    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"
Nov 02 14:11:18.710352 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.709915    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j" event={"ID":"4eb70645-1999-43ff-8177-e29cf03352b3","Type":"ContainerDied","Data":"1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407"}
Nov 02 14:11:18.710606 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.710456    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j" event={"ID":"4eb70645-1999-43ff-8177-e29cf03352b3","Type":"ContainerDied","Data":"222fdc5d5981069c7bb60f185fd2cb8a80bc85fa3a5f66a20f4247e37e72f598"}
Nov 02 14:11:18.710606 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.710513    6284 scope.go:122] "RemoveContainer" containerID="1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407"
Nov 02 14:11:18.724131 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.724099    6284 scope.go:122] "RemoveContainer" containerID="1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407"
Nov 02 14:11:18.724591 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:11:18.724566    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407\": container with ID starting with 1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407 not found: ID does not exist" containerID="1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407"
Nov 02 14:11:18.724700 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.724608    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407"} err="failed to get container status \"1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407\": rpc error: code = NotFound desc = could not find container \"1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407\": container with ID starting with 1bad8f34c27074b41e14ef297a4ab101600b11c62814b8e3f838e74bc6b0f407 not found: ID does not exist"
Nov 02 14:11:18.795404 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.795369    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-model-cache\") pod \"4eb70645-1999-43ff-8177-e29cf03352b3\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") "
Nov 02 14:11:18.795596 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.795470    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-tmp-dir\") pod \"4eb70645-1999-43ff-8177-e29cf03352b3\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") "
Nov 02 14:11:18.795596 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.795524    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb70645-1999-43ff-8177-e29cf03352b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb70645-1999-43ff-8177-e29cf03352b3-tls-certs\") pod \"4eb70645-1999-43ff-8177-e29cf03352b3\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") "
Nov 02 14:11:18.795596 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.795588    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-home\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-home\") pod \"4eb70645-1999-43ff-8177-e29cf03352b3\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") "
Nov 02 14:11:18.795765 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.795671    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-dshm\") pod \"4eb70645-1999-43ff-8177-e29cf03352b3\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") "
Nov 02 14:11:18.795765 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.795709    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-model-cache" pod "4eb70645-1999-43ff-8177-e29cf03352b3" (UID: "4eb70645-1999-43ff-8177-e29cf03352b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:18.795765 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.795758    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb70645-1999-43ff-8177-e29cf03352b3-kube-api-access-45qtz\" (UniqueName: \"kubernetes.io/projected/4eb70645-1999-43ff-8177-e29cf03352b3-kube-api-access-45qtz\") pod \"4eb70645-1999-43ff-8177-e29cf03352b3\" (UID: \"4eb70645-1999-43ff-8177-e29cf03352b3\") "
Nov 02 14:11:18.795929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.795750    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-tmp-dir" pod "4eb70645-1999-43ff-8177-e29cf03352b3" (UID: "4eb70645-1999-43ff-8177-e29cf03352b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:18.796010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.795926    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-home" pod "4eb70645-1999-43ff-8177-e29cf03352b3" (UID: "4eb70645-1999-43ff-8177-e29cf03352b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:18.796313 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.796289    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:18.796441 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.796315    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:18.796441 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.796329    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:18.797958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.797915    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb70645-1999-43ff-8177-e29cf03352b3-tls-certs" pod "4eb70645-1999-43ff-8177-e29cf03352b3" (UID: "4eb70645-1999-43ff-8177-e29cf03352b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:18.798082 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.797926    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-dshm" pod "4eb70645-1999-43ff-8177-e29cf03352b3" (UID: "4eb70645-1999-43ff-8177-e29cf03352b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:18.798082 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.797990    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb70645-1999-43ff-8177-e29cf03352b3-kube-api-access-45qtz" pod "4eb70645-1999-43ff-8177-e29cf03352b3" (UID: "4eb70645-1999-43ff-8177-e29cf03352b3"). InnerVolumeSpecName "kube-api-access-45qtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:18.897251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.897157    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb70645-1999-43ff-8177-e29cf03352b3-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:18.897251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.897190    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45qtz\" (UniqueName: \"kubernetes.io/projected/4eb70645-1999-43ff-8177-e29cf03352b3-kube-api-access-45qtz\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:18.897251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:18.897201    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb70645-1999-43ff-8177-e29cf03352b3-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:19.063095 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:19.063060    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"]
Nov 02 14:11:19.075380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:19.075343    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j"]
Nov 02 14:11:19.643615 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:19.643557    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-794667/ipp-route-794667-kserve-68689775d5-6bj4j" podUID="4eb70645-1999-43ff-8177-e29cf03352b3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:11:19.683001 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:19.682960    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:11:19.683171 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:19.683044    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:11:20.126574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.126528    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"]
Nov 02 14:11:20.127220 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.127177    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7" podUID="5db9013b-1cca-4f27-b7f7-a2ae7199e2bd" containerName="maas-api" containerID="cri-o://52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2" gracePeriod=30
Nov 02 14:11:20.179486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.179444    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb70645-1999-43ff-8177-e29cf03352b3" path="/var/lib/kubelet/pods/4eb70645-1999-43ff-8177-e29cf03352b3/volumes"
Nov 02 14:11:20.649760 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.622298    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 14:11:20.662552 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.662524    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.722622    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-kube-api-access-286th\" (UniqueName: \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-kube-api-access-286th\") pod \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.722671    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-credential-socket\") pod \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.722732    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-envoy\") pod \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.722773    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-maas-api-tls\") pod \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.722806    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-kube-api-access-8ntnm\" (UniqueName: \"kubernetes.io/projected/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-kube-api-access-8ntnm\") pod \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.722823    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-token\" (UniqueName: \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-token\") pod \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.722853    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-data\") pod \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.722910    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-podinfo\") pod \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.722997    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-metrics-certs\") pod \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.723112    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-certs\") pod \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.723156    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-openshift-service-ca\") pod \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\" (UID: \"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.723268    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istiod-ca-cert\") pod \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") "
Nov 02 14:11:20.724139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.723309    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-socket\") pod \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\" (UID: \"7a7b28fb-a02b-4f33-a5ce-0caf338b100c\") "
Nov 02 14:11:20.726139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.725646    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-kube-api-access-286th" pod "7a7b28fb-a02b-4f33-a5ce-0caf338b100c" (UID: "7a7b28fb-a02b-4f33-a5ce-0caf338b100c"). InnerVolumeSpecName "kube-api-access-286th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:20.726660 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.726610    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-openshift-service-ca" pod "5db9013b-1cca-4f27-b7f7-a2ae7199e2bd" (UID: "5db9013b-1cca-4f27-b7f7-a2ae7199e2bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:20.730156 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.730103    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-envoy" pod "7a7b28fb-a02b-4f33-a5ce-0caf338b100c" (UID: "7a7b28fb-a02b-4f33-a5ce-0caf338b100c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:20.732072 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.730240    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-kube-api-access-8ntnm" pod "5db9013b-1cca-4f27-b7f7-a2ae7199e2bd" (UID: "5db9013b-1cca-4f27-b7f7-a2ae7199e2bd"). InnerVolumeSpecName "kube-api-access-8ntnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:20.732252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.732138    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-metrics-certs" pod "5db9013b-1cca-4f27-b7f7-a2ae7199e2bd" (UID: "5db9013b-1cca-4f27-b7f7-a2ae7199e2bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:20.732928 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.732807    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-credential-socket" pod "7a7b28fb-a02b-4f33-a5ce-0caf338b100c" (UID: "7a7b28fb-a02b-4f33-a5ce-0caf338b100c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:20.732928 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.732824    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-certs" pod "7a7b28fb-a02b-4f33-a5ce-0caf338b100c" (UID: "7a7b28fb-a02b-4f33-a5ce-0caf338b100c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:20.733279 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.733236    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-socket" pod "7a7b28fb-a02b-4f33-a5ce-0caf338b100c" (UID: "7a7b28fb-a02b-4f33-a5ce-0caf338b100c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:20.733462 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.733428    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-maas-api-tls" pod "5db9013b-1cca-4f27-b7f7-a2ae7199e2bd" (UID: "5db9013b-1cca-4f27-b7f7-a2ae7199e2bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:20.735417 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.735383    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-podinfo" pod "7a7b28fb-a02b-4f33-a5ce-0caf338b100c" (UID: "7a7b28fb-a02b-4f33-a5ce-0caf338b100c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:11:20.736075 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.736039    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-data" pod "7a7b28fb-a02b-4f33-a5ce-0caf338b100c" (UID: "7a7b28fb-a02b-4f33-a5ce-0caf338b100c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:20.736075 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.736067    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istiod-ca-cert" pod "7a7b28fb-a02b-4f33-a5ce-0caf338b100c" (UID: "7a7b28fb-a02b-4f33-a5ce-0caf338b100c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:20.736290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.736270    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-token" pod "7a7b28fb-a02b-4f33-a5ce-0caf338b100c" (UID: "7a7b28fb-a02b-4f33-a5ce-0caf338b100c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:20.738386 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.738360    6284 generic.go:356] "Generic (PLEG): container finished" podID="5db9013b-1cca-4f27-b7f7-a2ae7199e2bd" containerID="52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2" exitCode=0
Nov 02 14:11:20.738481 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.738412    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"
Nov 02 14:11:20.738481 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.738442    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7" event={"ID":"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd","Type":"ContainerDied","Data":"52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2"}
Nov 02 14:11:20.738573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.738495    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7" event={"ID":"5db9013b-1cca-4f27-b7f7-a2ae7199e2bd","Type":"ContainerDied","Data":"2a533c49242bbf5fc5a91769754cc64f6a9415123907babb93f233f96ee676c2"}
Nov 02 14:11:20.738573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.738525    6284 scope.go:122] "RemoveContainer" containerID="52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2"
Nov 02 14:11:20.740879 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.740849    6284 generic.go:356] "Generic (PLEG): container finished" podID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerID="965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf" exitCode=0
Nov 02 14:11:20.741007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.740911    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"
Nov 02 14:11:20.741007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.740968    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" event={"ID":"7a7b28fb-a02b-4f33-a5ce-0caf338b100c","Type":"ContainerDied","Data":"965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf"}
Nov 02 14:11:20.741129 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.741024    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn" event={"ID":"7a7b28fb-a02b-4f33-a5ce-0caf338b100c","Type":"ContainerDied","Data":"d0610158797cbabe5451e98824e7acec1101f672aa94e06af85bf584098abacc"}
Nov 02 14:11:20.760250 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.757090    6284 scope.go:122] "RemoveContainer" containerID="52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2"
Nov 02 14:11:20.760554 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:11:20.760523    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2\": container with ID starting with 52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2 not found: ID does not exist" containerID="52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2"
Nov 02 14:11:20.762695 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.760593    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2"} err="failed to get container status \"52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2\": rpc error: code = NotFound desc = could not find container \"52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2\": container with ID starting with 52c206d7d25ce5f222d585f7b30d393f03381c77946c43b15c1e0511d94511b2 not found: ID does not exist"
Nov 02 14:11:20.762695 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.760624    6284 scope.go:122] "RemoveContainer" containerID="965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf"
Nov 02 14:11:20.776625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.776599    6284 scope.go:122] "RemoveContainer" containerID="965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf"
Nov 02 14:11:20.777066 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:11:20.777025    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf\": container with ID starting with 965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf not found: ID does not exist" containerID="965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf"
Nov 02 14:11:20.777180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.777078    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf"} err="failed to get container status \"965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf\": rpc error: code = NotFound desc = could not find container \"965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf\": container with ID starting with 965f5ded44d6b9be5c08a68dca0923d5ac0bacaaa92824772d5916634c214daf not found: ID does not exist"
Nov 02 14:11:20.803847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.803795    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"]
Nov 02 14:11:20.809117 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.809086    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cc8d39-648757446d-4pww7"]
Nov 02 14:11:20.825730 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825675    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.825730 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825708    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.825730 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825722    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.825730 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825738    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.826244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825751    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.826244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825762    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-286th\" (UniqueName: \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-kube-api-access-286th\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.826244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825773    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.826244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825785    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.826244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825800    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.826244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825813    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ntnm\" (UniqueName: \"kubernetes.io/projected/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd-kube-api-access-8ntnm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.826244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825823    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.826244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825834    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.826244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.825845    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a7b28fb-a02b-4f33-a5ce-0caf338b100c-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:20.843050 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.843016    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"]
Nov 02 14:11:20.847569 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:20.847533    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-vtpmn"]
Nov 02 14:11:22.169954 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:22.169915    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db9013b-1cca-4f27-b7f7-a2ae7199e2bd" path="/var/lib/kubelet/pods/5db9013b-1cca-4f27-b7f7-a2ae7199e2bd/volumes"
Nov 02 14:11:22.170506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:22.170489    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" path="/var/lib/kubelet/pods/7a7b28fb-a02b-4f33-a5ce-0caf338b100c/volumes"
Nov 02 14:11:37.621540 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.621496    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"]
Nov 02 14:11:37.622185 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.621776    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg" podUID="88d1ea2b-135f-4081-b93b-bf6018aa6931" containerName="maas-api" containerID="cri-o://e3e8dcc17342bf127b0d6257166df38f8fab66b5813b0eaae24c56544e083c2a" gracePeriod=30
Nov 02 14:11:37.859730 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.859690    6284 generic.go:356] "Generic (PLEG): container finished" podID="88d1ea2b-135f-4081-b93b-bf6018aa6931" containerID="e3e8dcc17342bf127b0d6257166df38f8fab66b5813b0eaae24c56544e083c2a" exitCode=0
Nov 02 14:11:37.859962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.859764    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg" event={"ID":"88d1ea2b-135f-4081-b93b-bf6018aa6931","Type":"ContainerDied","Data":"e3e8dcc17342bf127b0d6257166df38f8fab66b5813b0eaae24c56544e083c2a"}
Nov 02 14:11:37.938225 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.938013    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:11:37.965110 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.965071    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-maas-api-tls\") pod \"88d1ea2b-135f-4081-b93b-bf6018aa6931\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") "
Nov 02 14:11:37.965325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.965211    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d1ea2b-135f-4081-b93b-bf6018aa6931-kube-api-access-l8hz8\" (UniqueName: \"kubernetes.io/projected/88d1ea2b-135f-4081-b93b-bf6018aa6931-kube-api-access-l8hz8\") pod \"88d1ea2b-135f-4081-b93b-bf6018aa6931\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") "
Nov 02 14:11:37.965325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.965311    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-metrics-certs\") pod \"88d1ea2b-135f-4081-b93b-bf6018aa6931\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") "
Nov 02 14:11:37.965455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.965383    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d1ea2b-135f-4081-b93b-bf6018aa6931-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d1ea2b-135f-4081-b93b-bf6018aa6931-openshift-service-ca\") pod \"88d1ea2b-135f-4081-b93b-bf6018aa6931\" (UID: \"88d1ea2b-135f-4081-b93b-bf6018aa6931\") "
Nov 02 14:11:37.966169 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.966135    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d1ea2b-135f-4081-b93b-bf6018aa6931-openshift-service-ca" pod "88d1ea2b-135f-4081-b93b-bf6018aa6931" (UID: "88d1ea2b-135f-4081-b93b-bf6018aa6931"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:37.968187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.968142    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d1ea2b-135f-4081-b93b-bf6018aa6931-kube-api-access-l8hz8" pod "88d1ea2b-135f-4081-b93b-bf6018aa6931" (UID: "88d1ea2b-135f-4081-b93b-bf6018aa6931"). InnerVolumeSpecName "kube-api-access-l8hz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:37.968343 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.968319    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-metrics-certs" pod "88d1ea2b-135f-4081-b93b-bf6018aa6931" (UID: "88d1ea2b-135f-4081-b93b-bf6018aa6931"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:37.968480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:37.968359    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-maas-api-tls" pod "88d1ea2b-135f-4081-b93b-bf6018aa6931" (UID: "88d1ea2b-135f-4081-b93b-bf6018aa6931"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:38.067267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:38.067228    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:38.067267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:38.067262    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d1ea2b-135f-4081-b93b-bf6018aa6931-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:38.067267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:38.067271    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88d1ea2b-135f-4081-b93b-bf6018aa6931-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:38.067519 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:38.067280    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8hz8\" (UniqueName: \"kubernetes.io/projected/88d1ea2b-135f-4081-b93b-bf6018aa6931-kube-api-access-l8hz8\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:38.868709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:38.868684    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"
Nov 02 14:11:38.869165 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:38.868683    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg" event={"ID":"88d1ea2b-135f-4081-b93b-bf6018aa6931","Type":"ContainerDied","Data":"df4f768fecf84983b2fbcbc7ca581aa8bdcb3bfbdf76060d39be662c0b5b1c0b"}
Nov 02 14:11:38.869165 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:38.868812    6284 scope.go:122] "RemoveContainer" containerID="e3e8dcc17342bf127b0d6257166df38f8fab66b5813b0eaae24c56544e083c2a"
Nov 02 14:11:38.917871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:38.917810    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"]
Nov 02 14:11:38.926738 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:38.926705    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-13367a-787b69857c-wnwhg"]
Nov 02 14:11:40.169845 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:40.169810    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d1ea2b-135f-4081-b93b-bf6018aa6931" path="/var/lib/kubelet/pods/88d1ea2b-135f-4081-b93b-bf6018aa6931/volumes"
Nov 02 14:11:42.384964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:42.383765    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"]
Nov 02 14:11:42.384964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:42.384059    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy" containerID="cri-o://fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6" gracePeriod=30
Nov 02 14:11:42.390115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:42.388976    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"]
Nov 02 14:11:42.390115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:42.389211    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy" containerID="cri-o://37d2c6524eb0f6a8bd23d9a129cd9c1d7cb6062a82f72a0092bdb626357a8f24" gracePeriod=30
Nov 02 14:11:43.948259 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:11:43.948221    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:11:48.112117 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:48.112077    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:11:48.112735 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:48.112142    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:11:48.961400 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:48.961362    6284 generic.go:356] "Generic (PLEG): container finished" podID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerID="37d2c6524eb0f6a8bd23d9a129cd9c1d7cb6062a82f72a0092bdb626357a8f24" exitCode=0
Nov 02 14:11:48.961705 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:48.961531    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" event={"ID":"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f","Type":"ContainerDied","Data":"37d2c6524eb0f6a8bd23d9a129cd9c1d7cb6062a82f72a0092bdb626357a8f24"}
Nov 02 14:11:49.105212 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.105176    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 14:11:49.193067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.193036    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 14:11:49.279437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.279398    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-socket\") pod \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") "
Nov 02 14:11:49.279729 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.279708    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-credential-socket\") pod \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") "
Nov 02 14:11:49.279831 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.279753    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-token\" (UniqueName: \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-token\") pod \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") "
Nov 02 14:11:49.279831 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.279765    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-socket" pod "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" (UID: "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:49.279831 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.279803    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-data\") pod \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") "
Nov 02 14:11:49.280100 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.279859    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-envoy\") pod \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") "
Nov 02 14:11:49.280100 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.279888    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-certs\") pod \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") "
Nov 02 14:11:49.280100 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.279912    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istiod-ca-cert\") pod \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") "
Nov 02 14:11:49.280100 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.279968    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-podinfo\") pod \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") "
Nov 02 14:11:49.280100 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.280063    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-credential-socket" pod "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" (UID: "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:49.280337 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.280127    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-kube-api-access-6c4t6\" (UniqueName: \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-kube-api-access-6c4t6\") pod \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\" (UID: \"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f\") "
Nov 02 14:11:49.280689 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.280651    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-certs" pod "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" (UID: "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:49.281015 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.280998    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.281327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.281020    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.281327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.281036    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.281508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.281340    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istiod-ca-cert" pod "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" (UID: "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:49.282508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.282481    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-data" pod "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" (UID: "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:49.283036 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.283008    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-envoy" pod "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" (UID: "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:49.283678 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.283650    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-podinfo" pod "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" (UID: "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:11:49.284365 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.284345    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-kube-api-access-6c4t6" pod "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" (UID: "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f"). InnerVolumeSpecName "kube-api-access-6c4t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:49.284512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.284496    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-token" pod "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" (UID: "c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:49.382036 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.381897    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-envoy\") pod \"b21504bd-e77a-403a-ad00-7331870c5b59\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") "
Nov 02 14:11:49.382036 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382036    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-certs\") pod \"b21504bd-e77a-403a-ad00-7331870c5b59\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") "
Nov 02 14:11:49.382288 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382072    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-data\") pod \"b21504bd-e77a-403a-ad00-7331870c5b59\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") "
Nov 02 14:11:49.382288 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382120    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-credential-socket\") pod \"b21504bd-e77a-403a-ad00-7331870c5b59\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") "
Nov 02 14:11:49.382288 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382166    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-kube-api-access-7zsps\" (UniqueName: \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-kube-api-access-7zsps\") pod \"b21504bd-e77a-403a-ad00-7331870c5b59\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") "
Nov 02 14:11:49.382288 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382198    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-istio-token\" (UniqueName: \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-istio-token\") pod \"b21504bd-e77a-403a-ad00-7331870c5b59\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") "
Nov 02 14:11:49.382498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382320    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b21504bd-e77a-403a-ad00-7331870c5b59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21504bd-e77a-403a-ad00-7331870c5b59-istiod-ca-cert\") pod \"b21504bd-e77a-403a-ad00-7331870c5b59\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") "
Nov 02 14:11:49.382498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382347    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-certs" pod "b21504bd-e77a-403a-ad00-7331870c5b59" (UID: "b21504bd-e77a-403a-ad00-7331870c5b59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:49.382498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382415    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b21504bd-e77a-403a-ad00-7331870c5b59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21504bd-e77a-403a-ad00-7331870c5b59-istio-podinfo\") pod \"b21504bd-e77a-403a-ad00-7331870c5b59\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") "
Nov 02 14:11:49.382498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382443    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-data" pod "b21504bd-e77a-403a-ad00-7331870c5b59" (UID: "b21504bd-e77a-403a-ad00-7331870c5b59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:49.382498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382466    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-credential-socket" pod "b21504bd-e77a-403a-ad00-7331870c5b59" (UID: "b21504bd-e77a-403a-ad00-7331870c5b59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:49.382498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382473    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-socket\") pod \"b21504bd-e77a-403a-ad00-7331870c5b59\" (UID: \"b21504bd-e77a-403a-ad00-7331870c5b59\") "
Nov 02 14:11:49.382794 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.382750    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-socket" pod "b21504bd-e77a-403a-ad00-7331870c5b59" (UID: "b21504bd-e77a-403a-ad00-7331870c5b59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:49.383061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383035    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21504bd-e77a-403a-ad00-7331870c5b59-istiod-ca-cert" pod "b21504bd-e77a-403a-ad00-7331870c5b59" (UID: "b21504bd-e77a-403a-ad00-7331870c5b59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:49.383196 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383172    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.383196 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383196    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.383380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383211    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.383380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383226    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.383380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383238    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21504bd-e77a-403a-ad00-7331870c5b59-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.383380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383252    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.383380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383265    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c4t6\" (UniqueName: \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-kube-api-access-6c4t6\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.383380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383281    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.383380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383294    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.383380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383307    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.383380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.383322    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.384564 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.384531    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-envoy" pod "b21504bd-e77a-403a-ad00-7331870c5b59" (UID: "b21504bd-e77a-403a-ad00-7331870c5b59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:49.384764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.384734    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-istio-token" pod "b21504bd-e77a-403a-ad00-7331870c5b59" (UID: "b21504bd-e77a-403a-ad00-7331870c5b59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:49.384889 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.384872    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b21504bd-e77a-403a-ad00-7331870c5b59-istio-podinfo" pod "b21504bd-e77a-403a-ad00-7331870c5b59" (UID: "b21504bd-e77a-403a-ad00-7331870c5b59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:11:49.385116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.384918    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-kube-api-access-7zsps" pod "b21504bd-e77a-403a-ad00-7331870c5b59" (UID: "b21504bd-e77a-403a-ad00-7331870c5b59"). InnerVolumeSpecName "kube-api-access-7zsps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:49.484302 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.484260    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21504bd-e77a-403a-ad00-7331870c5b59-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.484302 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.484293    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21504bd-e77a-403a-ad00-7331870c5b59-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.484302 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.484303    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zsps\" (UniqueName: \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-kube-api-access-7zsps\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.484302 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.484314    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21504bd-e77a-403a-ad00-7331870c5b59-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:49.970963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.970913    6284 generic.go:356] "Generic (PLEG): container finished" podID="b21504bd-e77a-403a-ad00-7331870c5b59" containerID="fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6" exitCode=0
Nov 02 14:11:49.971161 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.970973    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"
Nov 02 14:11:49.971161 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.971000    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" event={"ID":"b21504bd-e77a-403a-ad00-7331870c5b59","Type":"ContainerDied","Data":"fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6"}
Nov 02 14:11:49.971161 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.971060    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn" event={"ID":"b21504bd-e77a-403a-ad00-7331870c5b59","Type":"ContainerDied","Data":"2e30c7ca53e149b9acec3c30d1c65040bffc493393d577fc073fbfa8d8ee4f96"}
Nov 02 14:11:49.971161 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.971090    6284 scope.go:122] "RemoveContainer" containerID="fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6"
Nov 02 14:11:49.973077 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.973045    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt" event={"ID":"c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f","Type":"ContainerDied","Data":"6ad31bb8bbaaa4eaf495334b686141f0d3c5aa2d317973e4477b426a4d2b78d6"}
Nov 02 14:11:49.973194 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.973160    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"
Nov 02 14:11:49.982039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.981997    6284 scope.go:122] "RemoveContainer" containerID="fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6"
Nov 02 14:11:49.982625 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:11:49.982601    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6\": container with ID starting with fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6 not found: ID does not exist" containerID="fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6"
Nov 02 14:11:49.982732 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.982638    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6"} err="failed to get container status \"fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6\": rpc error: code = NotFound desc = could not find container \"fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6\": container with ID starting with fd1b88bcd1f3560bd2817a6cf18e02b42aa8e7e2115f9d307a4ec64e6da6ccf6 not found: ID does not exist"
Nov 02 14:11:49.982732 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:49.982661    6284 scope.go:122] "RemoveContainer" containerID="37d2c6524eb0f6a8bd23d9a129cd9c1d7cb6062a82f72a0092bdb626357a8f24"
Nov 02 14:11:50.036698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:50.036658    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"]
Nov 02 14:11:50.046953 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:50.046903    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-hxxnn"]
Nov 02 14:11:50.086518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:50.086475    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"]
Nov 02 14:11:50.095997 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:50.095952    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzbxt"]
Nov 02 14:11:50.175801 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:50.175717    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" path="/var/lib/kubelet/pods/b21504bd-e77a-403a-ad00-7331870c5b59/volumes"
Nov 02 14:11:50.176373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:50.176354    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" path="/var/lib/kubelet/pods/c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f/volumes"
Nov 02 14:11:57.612191 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.612153    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"]
Nov 02 14:11:57.612659 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.612546    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn" podUID="bd924467-a73e-4ce4-b1ba-7843d4496790" containerName="maas-api" containerID="cri-o://5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd" gracePeriod=30
Nov 02 14:11:57.960989 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.960959    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:11:57.982361 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.982323    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd924467-a73e-4ce4-b1ba-7843d4496790-kube-api-access-wzdjs\" (UniqueName: \"kubernetes.io/projected/bd924467-a73e-4ce4-b1ba-7843d4496790-kube-api-access-wzdjs\") pod \"bd924467-a73e-4ce4-b1ba-7843d4496790\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") "
Nov 02 14:11:57.982574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.982376    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-metrics-certs\") pod \"bd924467-a73e-4ce4-b1ba-7843d4496790\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") "
Nov 02 14:11:57.982574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.982436    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd924467-a73e-4ce4-b1ba-7843d4496790-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd924467-a73e-4ce4-b1ba-7843d4496790-openshift-service-ca\") pod \"bd924467-a73e-4ce4-b1ba-7843d4496790\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") "
Nov 02 14:11:57.982574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.982541    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-maas-api-tls\") pod \"bd924467-a73e-4ce4-b1ba-7843d4496790\" (UID: \"bd924467-a73e-4ce4-b1ba-7843d4496790\") "
Nov 02 14:11:57.982957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.982900    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd924467-a73e-4ce4-b1ba-7843d4496790-openshift-service-ca" pod "bd924467-a73e-4ce4-b1ba-7843d4496790" (UID: "bd924467-a73e-4ce4-b1ba-7843d4496790"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:57.983115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.983092    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd924467-a73e-4ce4-b1ba-7843d4496790-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:57.984808 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.984764    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd924467-a73e-4ce4-b1ba-7843d4496790-kube-api-access-wzdjs" pod "bd924467-a73e-4ce4-b1ba-7843d4496790" (UID: "bd924467-a73e-4ce4-b1ba-7843d4496790"). InnerVolumeSpecName "kube-api-access-wzdjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:57.984964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.984800    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-maas-api-tls" pod "bd924467-a73e-4ce4-b1ba-7843d4496790" (UID: "bd924467-a73e-4ce4-b1ba-7843d4496790"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:57.985021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:57.984965    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-metrics-certs" pod "bd924467-a73e-4ce4-b1ba-7843d4496790" (UID: "bd924467-a73e-4ce4-b1ba-7843d4496790"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:58.093039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.092417    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:58.093039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.092454    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzdjs\" (UniqueName: \"kubernetes.io/projected/bd924467-a73e-4ce4-b1ba-7843d4496790-kube-api-access-wzdjs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:58.093039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.092470    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd924467-a73e-4ce4-b1ba-7843d4496790-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:11:58.112983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.108762    6284 generic.go:356] "Generic (PLEG): container finished" podID="bd924467-a73e-4ce4-b1ba-7843d4496790" containerID="5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd" exitCode=0
Nov 02 14:11:58.112983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.108928    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn" event={"ID":"bd924467-a73e-4ce4-b1ba-7843d4496790","Type":"ContainerDied","Data":"5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd"}
Nov 02 14:11:58.112983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.109043    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn" event={"ID":"bd924467-a73e-4ce4-b1ba-7843d4496790","Type":"ContainerDied","Data":"700f918bbacb7b1bce8eff2f50d8613dd7f552733d70a67bdf5369a9839c5469"}
Nov 02 14:11:58.112983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.109081    6284 scope.go:122] "RemoveContainer" containerID="5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd"
Nov 02 14:11:58.112983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.109295    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"
Nov 02 14:11:58.126190 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.126144    6284 scope.go:122] "RemoveContainer" containerID="5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd"
Nov 02 14:11:58.126654 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:11:58.126569    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd\": container with ID starting with 5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd not found: ID does not exist" containerID="5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd"
Nov 02 14:11:58.126654 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.126610    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd"} err="failed to get container status \"5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd\": rpc error: code = NotFound desc = could not find container \"5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd\": container with ID starting with 5805f355a9b9cd7e1ad5200ed79a42f51cb86a661a4d54b9caf3cb854e94cfcd not found: ID does not exist"
Nov 02 14:11:58.172870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.172837    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"]
Nov 02 14:11:58.173380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:58.173359    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-370905-d6547898c-6mcgn"]
Nov 02 14:11:59.886968 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:59.886916    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"]
Nov 02 14:11:59.887478 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:11:59.887156    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89" podUID="8bc08500-ee23-49bb-acb6-caa893c5b372" containerName="maas-api" containerID="cri-o://c2e16bc29170e5b75c1802ae13b9edc32cf13e0fe14ba4e6da3587017baaef43" gracePeriod=30
Nov 02 14:12:00.135598 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.135432    6284 generic.go:356] "Generic (PLEG): container finished" podID="8bc08500-ee23-49bb-acb6-caa893c5b372" containerID="c2e16bc29170e5b75c1802ae13b9edc32cf13e0fe14ba4e6da3587017baaef43" exitCode=0
Nov 02 14:12:00.135880 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.135861    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89" event={"ID":"8bc08500-ee23-49bb-acb6-caa893c5b372","Type":"ContainerDied","Data":"c2e16bc29170e5b75c1802ae13b9edc32cf13e0fe14ba4e6da3587017baaef43"}
Nov 02 14:12:00.177045 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.177002    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd924467-a73e-4ce4-b1ba-7843d4496790" path="/var/lib/kubelet/pods/bd924467-a73e-4ce4-b1ba-7843d4496790/volumes"
Nov 02 14:12:00.205162 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.205128    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:12:00.321037 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.320996    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc08500-ee23-49bb-acb6-caa893c5b372-kube-api-access-t2tpm\" (UniqueName: \"kubernetes.io/projected/8bc08500-ee23-49bb-acb6-caa893c5b372-kube-api-access-t2tpm\") pod \"8bc08500-ee23-49bb-acb6-caa893c5b372\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") "
Nov 02 14:12:00.321231 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.321064    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-maas-api-tls\") pod \"8bc08500-ee23-49bb-acb6-caa893c5b372\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") "
Nov 02 14:12:00.321231 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.321105    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc08500-ee23-49bb-acb6-caa893c5b372-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc08500-ee23-49bb-acb6-caa893c5b372-openshift-service-ca\") pod \"8bc08500-ee23-49bb-acb6-caa893c5b372\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") "
Nov 02 14:12:00.321356 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.321244    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-metrics-certs\") pod \"8bc08500-ee23-49bb-acb6-caa893c5b372\" (UID: \"8bc08500-ee23-49bb-acb6-caa893c5b372\") "
Nov 02 14:12:00.321559 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.321517    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc08500-ee23-49bb-acb6-caa893c5b372-openshift-service-ca" pod "8bc08500-ee23-49bb-acb6-caa893c5b372" (UID: "8bc08500-ee23-49bb-acb6-caa893c5b372"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:00.321975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.321920    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bc08500-ee23-49bb-acb6-caa893c5b372-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:12:00.323343 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.323312    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-metrics-certs" pod "8bc08500-ee23-49bb-acb6-caa893c5b372" (UID: "8bc08500-ee23-49bb-acb6-caa893c5b372"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:00.323436 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.323312    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc08500-ee23-49bb-acb6-caa893c5b372-kube-api-access-t2tpm" pod "8bc08500-ee23-49bb-acb6-caa893c5b372" (UID: "8bc08500-ee23-49bb-acb6-caa893c5b372"). InnerVolumeSpecName "kube-api-access-t2tpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:00.323436 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.323402    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-maas-api-tls" pod "8bc08500-ee23-49bb-acb6-caa893c5b372" (UID: "8bc08500-ee23-49bb-acb6-caa893c5b372"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:00.422988 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.422950    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:12:00.422988 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.422981    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bc08500-ee23-49bb-acb6-caa893c5b372-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:12:00.423222 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:00.423014    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2tpm\" (UniqueName: \"kubernetes.io/projected/8bc08500-ee23-49bb-acb6-caa893c5b372-kube-api-access-t2tpm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:12:01.145057 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:01.145027    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"
Nov 02 14:12:01.145576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:01.145013    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89" event={"ID":"8bc08500-ee23-49bb-acb6-caa893c5b372","Type":"ContainerDied","Data":"b40a31bdf5c31ab5c2a2c772de4a3dcdbfa4885190804ac36654f9c6486c8cb4"}
Nov 02 14:12:01.145576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:01.145183    6284 scope.go:122] "RemoveContainer" containerID="c2e16bc29170e5b75c1802ae13b9edc32cf13e0fe14ba4e6da3587017baaef43"
Nov 02 14:12:01.192651 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:01.192611    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"]
Nov 02 14:12:01.202437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:01.202345    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-de493d-d9c4ff544-9nm89"]
Nov 02 14:12:02.170293 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:02.170256    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc08500-ee23-49bb-acb6-caa893c5b372" path="/var/lib/kubelet/pods/8bc08500-ee23-49bb-acb6-caa893c5b372/volumes"
Nov 02 14:12:12.505142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.505098    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"]
Nov 02 14:12:12.505864 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.505843    6284 cpu_manager.go:414] "containerMap: removing container" podUID="8bc08500-ee23-49bb-acb6-caa893c5b372" containerName="maas-api"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.505867    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc08500-ee23-49bb-acb6-caa893c5b372" containerName="maas-api"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.505887    6284 cpu_manager.go:414] "containerMap: removing container" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.505897    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.505920    6284 cpu_manager.go:414] "containerMap: removing container" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.505929    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.505997    6284 cpu_manager.go:414] "containerMap: removing container" podUID="5db9013b-1cca-4f27-b7f7-a2ae7199e2bd" containerName="maas-api"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506005    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db9013b-1cca-4f27-b7f7-a2ae7199e2bd" containerName="maas-api"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506018    6284 cpu_manager.go:414] "containerMap: removing container" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506026    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506043    6284 cpu_manager.go:414] "containerMap: removing container" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy"
Nov 02 14:12:12.506061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506050    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506072    6284 cpu_manager.go:414] "containerMap: removing container" podUID="658fde51-0a39-44e5-974c-1a1deeb86b47" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506080    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="658fde51-0a39-44e5-974c-1a1deeb86b47" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506098    6284 cpu_manager.go:414] "containerMap: removing container" podUID="4eb70645-1999-43ff-8177-e29cf03352b3" containerName="main"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506106    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb70645-1999-43ff-8177-e29cf03352b3" containerName="main"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506122    6284 cpu_manager.go:414] "containerMap: removing container" podUID="bd924467-a73e-4ce4-b1ba-7843d4496790" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506129    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd924467-a73e-4ce4-b1ba-7843d4496790" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506144    6284 cpu_manager.go:414] "containerMap: removing container" podUID="88d1ea2b-135f-4081-b93b-bf6018aa6931" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506151    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d1ea2b-135f-4081-b93b-bf6018aa6931" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506169    6284 cpu_manager.go:414] "containerMap: removing container" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506178    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506325    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d1ea2b-135f-4081-b93b-bf6018aa6931" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506337    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd924467-a73e-4ce4-b1ba-7843d4496790" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506348    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7b28fb-a02b-4f33-a5ce-0caf338b100c" containerName="istio-proxy"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506360    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db9013b-1cca-4f27-b7f7-a2ae7199e2bd" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506371    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="112fb288-71fc-4554-a30d-2f7776fb51ea" containerName="istio-proxy"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506384    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc08500-ee23-49bb-acb6-caa893c5b372" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506394    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="323cdaf7-0099-49fe-ac81-d64d7acd09c0" containerName="istio-proxy"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506403    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21504bd-e77a-403a-ad00-7331870c5b59" containerName="istio-proxy"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506414    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="658fde51-0a39-44e5-974c-1a1deeb86b47" containerName="maas-api"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506423    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb70645-1999-43ff-8177-e29cf03352b3" containerName="main"
Nov 02 14:12:12.506599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.506433    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6abbb9f-1bcc-4ec1-bff5-6a727c898e4f" containerName="istio-proxy"
Nov 02 14:12:12.512108 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.512079    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.542137 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.538719    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"]
Nov 02 14:12:12.578106 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.578061    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.578283 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.578128    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.578353 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.578315    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.578399 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.578364    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t9qv\" (UniqueName: \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-kube-api-access-2t9qv\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.578489 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.578467    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.578540 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.578526    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.578575 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.578567    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.578655 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.578642    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.578719 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.578706    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.680329 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680277    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.680329 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680333    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t9qv\" (UniqueName: \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-kube-api-access-2t9qv\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.680601 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680372    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.680601 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680423    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.680601 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680476    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.680601 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680532    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.680829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680611    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.680829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680666    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.680829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680713    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.680829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680791    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.681122 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.680913    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.681186 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.681134    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.681186 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.681173    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.681333 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.681307    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.682866 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.682844    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.683164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.683141    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.705355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.705316    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t9qv\" (UniqueName: \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-kube-api-access-2t9qv\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.707448 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.707416    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:12.828322 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:12.828224    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:13.202770 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:13.202720    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"]
Nov 02 14:12:13.204349 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:12:13.204316    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24249a3_c205_40cf_b9d4_87dc53b97ff8.slice/crio-4bb893afe940fb24b91bbd2af97e281c2fab2276307c5244684da3fc75fc0c5e WatchSource:0}: Error finding container 4bb893afe940fb24b91bbd2af97e281c2fab2276307c5244684da3fc75fc0c5e: Status 404 returned error can't find the container with id 4bb893afe940fb24b91bbd2af97e281c2fab2276307c5244684da3fc75fc0c5e
Nov 02 14:12:13.207129 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:13.207098    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:12:13.207244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:13.207166    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:12:13.207244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:13.207197    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:12:13.221879 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:13.221847    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" event={"ID":"c24249a3-c205-40cf-b9d4-87dc53b97ff8","Type":"ContainerStarted","Data":"4bb893afe940fb24b91bbd2af97e281c2fab2276307c5244684da3fc75fc0c5e"}
Nov 02 14:12:13.895450 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:12:13.895400    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:12:14.230382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:14.230285    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" event={"ID":"c24249a3-c205-40cf-b9d4-87dc53b97ff8","Type":"ContainerStarted","Data":"6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51"}
Nov 02 14:12:14.301446 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:14.293235    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" podStartSLOduration=2.293217134 podStartE2EDuration="2.293217134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:14.282676182 +0000 UTC m=+1666.242051814" watchObservedRunningTime="2026-11-02 14:12:14.293217134 +0000 UTC m=+1666.252592743"
Nov 02 14:12:14.828903 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:14.828867    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:14.830512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:14.830481    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 14:12:14.830648 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:14.830560    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 14:12:15.828834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:15.828792    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 14:12:15.829405 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:15.828860    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 14:12:16.829681 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:16.829640    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 14:12:16.830230 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:16.829732    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 14:12:17.829191 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:17.829154    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 14:12:17.829394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:17.829212    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 14:12:18.829083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:18.829047    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 14:12:18.829513 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:18.829108    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 14:12:19.829156 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:19.829121    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 02 14:12:19.829784 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:19.829179    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 02 14:12:21.363636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:21.363592    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:21.364201 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:21.364078    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:21.364878 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:21.364856    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:12:43.895692 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:12:43.895649    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:12:57.661532 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.661486    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"]
Nov 02 14:12:57.666398 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.666372    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.694273 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.694234    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"]
Nov 02 14:12:57.787347 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.787294    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.787742 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.787715    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.788050 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.788026    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.788330 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.788311    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d853398-09b5-474a-967f-bc2a47e862c3-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.788589 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.788559    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d853398-09b5-474a-967f-bc2a47e862c3-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.788815 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.788798    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.789031 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.789005    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.789246 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.789225    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmf6b\" (UniqueName: \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-kube-api-access-tmf6b\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.789514 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.789496    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.890594 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.890563    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.890810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.890613    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.890810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.890651    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.890810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.890690    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.890810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.890736    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d853398-09b5-474a-967f-bc2a47e862c3-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.890810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.890769    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d853398-09b5-474a-967f-bc2a47e862c3-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.891147 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.890828    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.891147 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.890859    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.891147 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.890883    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmf6b\" (UniqueName: \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-kube-api-access-tmf6b\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.891328 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.891268    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.891328 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.891313    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.891827 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.891457    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.891827 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.891517    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.891827 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.891735    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d853398-09b5-474a-967f-bc2a47e862c3-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.893715 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.893673    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.893920 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.893899    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d853398-09b5-474a-967f-bc2a47e862c3-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.919130 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.919043    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.919304 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.919275    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmf6b\" (UniqueName: \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-kube-api-access-tmf6b\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:57.982482 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:57.982443    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:58.315872 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:58.315844    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"]
Nov 02 14:12:58.318402 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:12:58.318372    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d853398_09b5_474a_967f_bc2a47e862c3.slice/crio-44719ecd2e4a23177da4a779a2c048b431b22aefe4a7b3f2d800f74de1f6ee97 WatchSource:0}: Error finding container 44719ecd2e4a23177da4a779a2c048b431b22aefe4a7b3f2d800f74de1f6ee97: Status 404 returned error can't find the container with id 44719ecd2e4a23177da4a779a2c048b431b22aefe4a7b3f2d800f74de1f6ee97
Nov 02 14:12:58.321143 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:58.321103    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:12:58.321235 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:58.321174    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:12:58.321235 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:58.321206    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:12:58.584007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:58.583885    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" event={"ID":"6d853398-09b5-474a-967f-bc2a47e862c3","Type":"ContainerStarted","Data":"116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3"}
Nov 02 14:12:58.584007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:58.583962    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" event={"ID":"6d853398-09b5-474a-967f-bc2a47e862c3","Type":"ContainerStarted","Data":"44719ecd2e4a23177da4a779a2c048b431b22aefe4a7b3f2d800f74de1f6ee97"}
Nov 02 14:12:58.630154 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:58.630073    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" podStartSLOduration=1.630049664 podStartE2EDuration="1.630049664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:58.627003626 +0000 UTC m=+1710.586379234" watchObservedRunningTime="2026-11-02 14:12:58.630049664 +0000 UTC m=+1710.589425271"
Nov 02 14:12:58.983135 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:58.983027    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:12:59.985437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:59.985396    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:12:59.985960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:12:59.985490    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:13:00.982949 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:00.982894    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:13:00.983175 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:00.982977    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:13:01.986109 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:01.986071    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:13:01.986593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:01.986133    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:13:02.983611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:02.983564    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:13:02.983842 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:02.983630    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:13:03.983974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:03.983899    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:13:03.984493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:03.984008    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:13:04.983472 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:04.983429    6284 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 14:13:04.983701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:04.983490    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 14:13:05.988025 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:05.987992    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:13:05.990423 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:05.988276    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:13:05.990423 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:05.989193    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:13:13.926058 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:13:13.926013    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:13:31.789026 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:31.788984    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"]
Nov 02 14:13:31.789587 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:31.789279    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r" podUID="e2454243-a80d-4b6d-9602-85036725c347" containerName="maas-api" containerID="cri-o://6c125922fde4fa6cd1f6d5209a93ac967205a5d51879b92893e43d319b7470ec" gracePeriod=30
Nov 02 14:13:31.915293 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:31.915260    6284 generic.go:356] "Generic (PLEG): container finished" podID="e2454243-a80d-4b6d-9602-85036725c347" containerID="6c125922fde4fa6cd1f6d5209a93ac967205a5d51879b92893e43d319b7470ec" exitCode=0
Nov 02 14:13:31.915500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:31.915341    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r" event={"ID":"e2454243-a80d-4b6d-9602-85036725c347","Type":"ContainerDied","Data":"6c125922fde4fa6cd1f6d5209a93ac967205a5d51879b92893e43d319b7470ec"}
Nov 02 14:13:31.915500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:31.915428    6284 scope.go:122] "RemoveContainer" containerID="48df3d41908aa966b3fcb13b141e4036eef092fc0c4854e832696144ff65a434"
Nov 02 14:13:32.169990 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.169170    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:13:32.351000 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.350446    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-metrics-certs\") pod \"e2454243-a80d-4b6d-9602-85036725c347\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") "
Nov 02 14:13:32.351403 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.351381    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2454243-a80d-4b6d-9602-85036725c347-kube-api-access-hf292\" (UniqueName: \"kubernetes.io/projected/e2454243-a80d-4b6d-9602-85036725c347-kube-api-access-hf292\") pod \"e2454243-a80d-4b6d-9602-85036725c347\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") "
Nov 02 14:13:32.351707 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.351690    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2454243-a80d-4b6d-9602-85036725c347-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2454243-a80d-4b6d-9602-85036725c347-openshift-service-ca\") pod \"e2454243-a80d-4b6d-9602-85036725c347\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") "
Nov 02 14:13:32.352389 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.352117    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2454243-a80d-4b6d-9602-85036725c347-openshift-service-ca" pod "e2454243-a80d-4b6d-9602-85036725c347" (UID: "e2454243-a80d-4b6d-9602-85036725c347"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:32.352646 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.352629    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-maas-api-tls\") pod \"e2454243-a80d-4b6d-9602-85036725c347\" (UID: \"e2454243-a80d-4b6d-9602-85036725c347\") "
Nov 02 14:13:32.354917 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.354895    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2454243-a80d-4b6d-9602-85036725c347-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:32.355608 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.355570    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-metrics-certs" pod "e2454243-a80d-4b6d-9602-85036725c347" (UID: "e2454243-a80d-4b6d-9602-85036725c347"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:32.358001 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.357922    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-maas-api-tls" pod "e2454243-a80d-4b6d-9602-85036725c347" (UID: "e2454243-a80d-4b6d-9602-85036725c347"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:32.360618 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.360572    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2454243-a80d-4b6d-9602-85036725c347-kube-api-access-hf292" pod "e2454243-a80d-4b6d-9602-85036725c347" (UID: "e2454243-a80d-4b6d-9602-85036725c347"). InnerVolumeSpecName "kube-api-access-hf292". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:32.456579 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.456482    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:32.456579 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.456519    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2454243-a80d-4b6d-9602-85036725c347-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:32.456579 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.456530    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf292\" (UniqueName: \"kubernetes.io/projected/e2454243-a80d-4b6d-9602-85036725c347-kube-api-access-hf292\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:32.924471 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.924437    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"
Nov 02 14:13:32.924471 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.924469    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r" event={"ID":"e2454243-a80d-4b6d-9602-85036725c347","Type":"ContainerDied","Data":"aff95199aa348e2901d996e9aef9f2edd94ac8920daa5ed292f2a522c2b31012"}
Nov 02 14:13:32.925029 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.924524    6284 scope.go:122] "RemoveContainer" containerID="6c125922fde4fa6cd1f6d5209a93ac967205a5d51879b92893e43d319b7470ec"
Nov 02 14:13:32.978804 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.978765    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"]
Nov 02 14:13:32.987676 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:32.987641    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8acd34-844457576f-csr5r"]
Nov 02 14:13:34.170436 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:34.170400    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2454243-a80d-4b6d-9602-85036725c347" path="/var/lib/kubelet/pods/e2454243-a80d-4b6d-9602-85036725c347/volumes"
Nov 02 14:13:38.605430 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.605388    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"]
Nov 02 14:13:38.615989 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.606818    6284 cpu_manager.go:414] "containerMap: removing container" podUID="e2454243-a80d-4b6d-9602-85036725c347" containerName="maas-api"
Nov 02 14:13:38.616431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.616402    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2454243-a80d-4b6d-9602-85036725c347" containerName="maas-api"
Nov 02 14:13:38.616587 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.616574    6284 cpu_manager.go:414] "containerMap: removing container" podUID="e2454243-a80d-4b6d-9602-85036725c347" containerName="maas-api"
Nov 02 14:13:38.616673 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.616662    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2454243-a80d-4b6d-9602-85036725c347" containerName="maas-api"
Nov 02 14:13:38.617123 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.617103    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2454243-a80d-4b6d-9602-85036725c347" containerName="maas-api"
Nov 02 14:13:38.617249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.617238    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2454243-a80d-4b6d-9602-85036725c347" containerName="maas-api"
Nov 02 14:13:38.623003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.622971    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.673609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.673447    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"]
Nov 02 14:13:38.701011 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.700967    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cdaf64-openshift-default-dockercfg-g8696\""
Nov 02 14:13:38.732000 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.731924    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.732324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.732039    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.732324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.732079    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnpxf\" (UniqueName: \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-kube-api-access-mnpxf\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.732324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.732100    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.732324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.732125    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.732574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.732423    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.732574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.732491    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.732660 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.732596    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca316e7-a65a-4964-a3f9-73b192cb285f-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.732660 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.732652    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.833539 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.833497    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.833744 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.833585    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.833744 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.833654    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.833744 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.833699    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnpxf\" (UniqueName: \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-kube-api-access-mnpxf\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.833744 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.833730    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.833996 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.833871    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.834075 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.834055    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.834132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.834090    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.834186 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.834158    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca316e7-a65a-4964-a3f9-73b192cb285f-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.834239 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.834176    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.834339 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.834309    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.834408 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.834390    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.834470 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.834450    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.834979 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.834919    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca316e7-a65a-4964-a3f9-73b192cb285f-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.836181 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.836162    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.836295 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.836192    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.858265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.858198    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:38.858423 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:38.858325    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnpxf\" (UniqueName: \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-kube-api-access-mnpxf\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:39.006374 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:39.006332    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:39.363718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:39.363681    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"]
Nov 02 14:13:39.364867 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:13:39.364839    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca316e7_a65a_4964_a3f9_73b192cb285f.slice/crio-98ac53be24ee0519e64a46f982fc832d7dd28444cfb92578601edcb2b1e039b6 WatchSource:0}: Error finding container 98ac53be24ee0519e64a46f982fc832d7dd28444cfb92578601edcb2b1e039b6: Status 404 returned error can't find the container with id 98ac53be24ee0519e64a46f982fc832d7dd28444cfb92578601edcb2b1e039b6
Nov 02 14:13:39.367518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:39.367483    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:13:39.367608 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:39.367565    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:13:39.367608 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:39.367595    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:13:39.978599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:39.978561    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" event={"ID":"6ca316e7-a65a-4964-a3f9-73b192cb285f","Type":"ContainerStarted","Data":"7f3bebce1cd32d74025c7ade5894d2635c8853fe6fbaf6f6ae3bd26d55a65bb5"}
Nov 02 14:13:39.979663 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:39.978626    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" event={"ID":"6ca316e7-a65a-4964-a3f9-73b192cb285f","Type":"ContainerStarted","Data":"98ac53be24ee0519e64a46f982fc832d7dd28444cfb92578601edcb2b1e039b6"}
Nov 02 14:13:40.007385 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:40.007349    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:40.008901 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:40.008876    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 14:13:40.009041 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:40.008950    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 14:13:40.026080 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:40.026017    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podStartSLOduration=2.025998955 podStartE2EDuration="2.025998955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:40.023325686 +0000 UTC m=+1751.982701292" watchObservedRunningTime="2026-11-02 14:13:40.025998955 +0000 UTC m=+1751.985374620"
Nov 02 14:13:41.007202 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:41.007156    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 14:13:41.008224 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:41.007236    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 14:13:42.007380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:42.007336    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 14:13:42.007804 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:42.007435    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 14:13:43.007478 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:43.007442    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 14:13:43.007913 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:43.007523    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 14:13:44.004199 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:13:44.004155    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:13:44.011436 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:44.011399    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 14:13:44.012148 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:44.012116    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 14:13:45.017993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:45.014981    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 14:13:45.017993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:45.015047    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 14:13:45.620960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:45.596120    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 14:13:46.010410 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:46.010371    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 14:13:46.011441 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:46.011404    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 14:13:47.065455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:47.065400    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 14:13:47.066089 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:47.066052    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 14:13:48.013070 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:48.013040    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:48.013373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:48.013347    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:48.014359 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:48.014337    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:13:49.328081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:49.328043    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"]
Nov 02 14:13:49.328768 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:49.328674    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy" containerID="cri-o://4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836" gracePeriod=30
Nov 02 14:13:51.072893 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.072852    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"]
Nov 02 14:13:51.073814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.073769    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj" podUID="b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9" containerName="maas-api" containerID="cri-o://ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a" gracePeriod=30
Nov 02 14:13:51.378219 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.378191    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:13:51.398954 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.398901    6284 generic.go:356] "Generic (PLEG): container finished" podID="b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9" containerID="ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a" exitCode=0
Nov 02 14:13:51.399164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.398968    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"
Nov 02 14:13:51.399164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.398986    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj" event={"ID":"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9","Type":"ContainerDied","Data":"ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a"}
Nov 02 14:13:51.399164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.399049    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj" event={"ID":"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9","Type":"ContainerDied","Data":"bc19a18a0291c75ca0d38d2d4aae559595d72b30482ce89cdef3e8550fa3a40e"}
Nov 02 14:13:51.399164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.399079    6284 scope.go:122] "RemoveContainer" containerID="ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a"
Nov 02 14:13:51.414547 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.414520    6284 scope.go:122] "RemoveContainer" containerID="ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a"
Nov 02 14:13:51.415092 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:13:51.415062    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a\": container with ID starting with ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a not found: ID does not exist" containerID="ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a"
Nov 02 14:13:51.415214 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.415099    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a"} err="failed to get container status \"ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a\": rpc error: code = NotFound desc = could not find container \"ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a\": container with ID starting with ded7e066b5d505e153c18666f30f421c48a91ed51816a3a5cde01be0ddb1425a not found: ID does not exist"
Nov 02 14:13:51.498853 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.498809    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-kube-api-access-f4xjh\" (UniqueName: \"kubernetes.io/projected/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-kube-api-access-f4xjh\") pod \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") "
Nov 02 14:13:51.499111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.498970    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-metrics-certs\") pod \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") "
Nov 02 14:13:51.499111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.499014    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-openshift-service-ca\") pod \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") "
Nov 02 14:13:51.499111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.499035    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-maas-api-tls\") pod \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\" (UID: \"b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9\") "
Nov 02 14:13:51.499429 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.499401    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-openshift-service-ca" pod "b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9" (UID: "b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:51.499533 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.499515    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:51.501324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.501287    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-maas-api-tls" pod "b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9" (UID: "b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:51.501324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.501306    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-kube-api-access-f4xjh" pod "b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9" (UID: "b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9"). InnerVolumeSpecName "kube-api-access-f4xjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:51.501521 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.501374    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-metrics-certs" pod "b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9" (UID: "b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:51.619041 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.616443    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:51.619041 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.616479    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:51.619041 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.616494    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xjh\" (UniqueName: \"kubernetes.io/projected/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9-kube-api-access-f4xjh\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:51.747631 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.747589    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"]
Nov 02 14:13:51.755323 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:51.755282    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-732d3a-7c4d9f549-j7lbj"]
Nov 02 14:13:52.170001 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:52.169919    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9" path="/var/lib/kubelet/pods/b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9/volumes"
Nov 02 14:13:54.601998 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.601907    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 14:13:54.756090 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756045    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-certs\") pod \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") "
Nov 02 14:13:54.756274 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756114    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istiod-ca-cert\") pod \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") "
Nov 02 14:13:54.756274 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756191    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-socket\") pod \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") "
Nov 02 14:13:54.756274 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756213    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-podinfo\") pod \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") "
Nov 02 14:13:54.756470 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756368    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-kube-api-access-jz8z2\" (UniqueName: \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-kube-api-access-jz8z2\") pod \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") "
Nov 02 14:13:54.756470 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756442    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-data\") pod \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") "
Nov 02 14:13:54.756584 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756541    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-credential-socket\") pod \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") "
Nov 02 14:13:54.756584 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756436    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-certs" pod "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" (UID: "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:54.756701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756575    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-socket" pod "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" (UID: "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:54.756701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756588    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-envoy\") pod \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") "
Nov 02 14:13:54.756701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756654    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-data" pod "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" (UID: "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:54.756701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756686    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istiod-ca-cert" pod "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" (UID: "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:54.756901 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756730    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-token\" (UniqueName: \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-token\") pod \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\" (UID: \"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956\") "
Nov 02 14:13:54.756901 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.756823    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-credential-socket" pod "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" (UID: "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:54.757373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.757347    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.757500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.757380    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.757500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.757399    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.757500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.757415    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.757500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.757429    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.758691 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.758667    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-envoy" pod "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" (UID: "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:54.758691 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.758674    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-podinfo" pod "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" (UID: "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:13:54.758857 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.758761    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-kube-api-access-jz8z2" pod "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" (UID: "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956"). InnerVolumeSpecName "kube-api-access-jz8z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:54.758912 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.758900    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-token" pod "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" (UID: "e4c8dfe1-d27b-44d9-8c8b-757ad73ae956"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:54.858967 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.858845    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.858967 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.858877    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.858967 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.858890    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.858967 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:54.858901    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz8z2\" (UniqueName: \"kubernetes.io/projected/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956-kube-api-access-jz8z2\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:13:55.434526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:55.432561    6284 generic.go:356] "Generic (PLEG): container finished" podID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerID="4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836" exitCode=0
Nov 02 14:13:55.434526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:55.432739    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" event={"ID":"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956","Type":"ContainerDied","Data":"4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836"}
Nov 02 14:13:55.434526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:55.432813    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c" event={"ID":"e4c8dfe1-d27b-44d9-8c8b-757ad73ae956","Type":"ContainerDied","Data":"00ec23cb9f7a55a70ea967c548310162b1bb259e5f9e260d5d2d41a69ba9086f"}
Nov 02 14:13:55.434526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:55.432862    6284 scope.go:122] "RemoveContainer" containerID="4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836"
Nov 02 14:13:55.434526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:55.433077    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"
Nov 02 14:13:55.466394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:55.462138    6284 scope.go:122] "RemoveContainer" containerID="4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836"
Nov 02 14:13:55.466394 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:13:55.462608    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836\": container with ID starting with 4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836 not found: ID does not exist" containerID="4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836"
Nov 02 14:13:55.466394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:55.462649    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836"} err="failed to get container status \"4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836\": rpc error: code = NotFound desc = could not find container \"4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836\": container with ID starting with 4f1bcc220eb0afb07c7a7d22ea2d614161a1542d393474334411d970876a9836 not found: ID does not exist"
Nov 02 14:13:55.522967 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:55.521304    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"]
Nov 02 14:13:55.541973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:55.539024    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9zv5c"]
Nov 02 14:13:56.170967 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:13:56.170915    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" path="/var/lib/kubelet/pods/e4c8dfe1-d27b-44d9-8c8b-757ad73ae956/volumes"
Nov 02 14:14:01.662659 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.662611    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"]
Nov 02 14:14:01.663383 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.663363    6284 cpu_manager.go:414] "containerMap: removing container" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy"
Nov 02 14:14:01.663430 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.663384    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy"
Nov 02 14:14:01.663430 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.663397    6284 cpu_manager.go:414] "containerMap: removing container" podUID="b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9" containerName="maas-api"
Nov 02 14:14:01.663430 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.663402    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9" containerName="maas-api"
Nov 02 14:14:01.663573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.663489    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c8dfe1-d27b-44d9-8c8b-757ad73ae956" containerName="istio-proxy"
Nov 02 14:14:01.663573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.663497    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c9b9a4-16a1-434e-8e6c-4eaa74c599e9" containerName="maas-api"
Nov 02 14:14:01.666950 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.666916    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.701873 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.700776    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"]
Nov 02 14:14:01.853158 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.853119    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.853158 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.853161    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.853428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.853182    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.853428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.853353    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v478l\" (UniqueName: \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-kube-api-access-v478l\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.853517 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.853485    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.853562 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.853519    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.853670 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.853646    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.853800 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.853694    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.853859 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.853808    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.954893    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.954984    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.955043    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.955101    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.955121    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.955245    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955476 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.955294    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v478l\" (UniqueName: \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-kube-api-access-v478l\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955476 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.955423    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955476 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.955462    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.955479    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.955848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.955824    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.956053 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.956028    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.956053 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.956043    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.956226 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.956075    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.957609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.957582    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.957733 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.957718    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.976225 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.976186    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.976745 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.976720    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v478l\" (UniqueName: \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-kube-api-access-v478l\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:01.980639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:01.980612    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:02.321894 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:02.321860    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"]
Nov 02 14:14:02.331499 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:02.331392    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:02.331658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:02.331557    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:02.331729 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:02.331671    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:02.485487 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:02.485452    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" event={"ID":"4e13ec12-9d79-4a64-ba6d-914250c86d2d","Type":"ContainerStarted","Data":"4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6"}
Nov 02 14:14:02.485679 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:02.485517    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" event={"ID":"4e13ec12-9d79-4a64-ba6d-914250c86d2d","Type":"ContainerStarted","Data":"20523697a5e40ee86ff349a2d872ede4174d911a94b17040eb5b3a30d669e94a"}
Nov 02 14:14:02.531230 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:02.531160    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" podStartSLOduration=1.5311368170000001 podStartE2EDuration="1.531136817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:02.526986311 +0000 UTC m=+1774.486361915" watchObservedRunningTime="2026-11-02 14:14:02.531136817 +0000 UTC m=+1774.490512426"
Nov 02 14:14:02.981306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:02.981262    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:03.983255 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:03.983215    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 14:14:03.983742 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:03.983282    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 14:14:04.990895 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:04.990858    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 14:14:04.991508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:04.991463    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 14:14:05.981698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:05.981657    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 14:14:05.981895 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:05.981713    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 14:14:06.986960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:06.984555    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 14:14:06.986960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:06.984629    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 14:14:07.988962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:07.983430    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 14:14:07.988962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:07.983499    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 14:14:08.294182 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:08.294071    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:14:08.378750 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:08.378106    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:14:08.615022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:08.612428    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:14:08.982436 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:08.982342    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 14:14:08.982810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:08.982743    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 14:14:09.294655 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.294608    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 02 14:14:09.298436 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.297560    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:14:09.456383 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.378421    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:14:09.456383 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.379198    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:14:09.552711 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.552592    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:14:09.575134 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.553629    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:14:09.623034 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.615045    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 02 14:14:09.652003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.628539    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 14:14:09.773582 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.771693    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 14:14:09.982526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.982425    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 14:14:09.982526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:09.982503    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 14:14:11.412812 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:11.412776    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:11.413485 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:11.413216    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:11.414068 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:11.414050    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:14:13.914354 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:14:13.912820    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:14:14.673863 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.673821    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"]
Nov 02 14:14:14.679379 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.679353    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.693921 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.693894    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-2cddf2-serving-cert\""
Nov 02 14:14:14.707914 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.707860    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"]
Nov 02 14:14:14.824290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.824254    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drnth\" (UniqueName: \"kubernetes.io/projected/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-kube-api-access-drnth\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.824290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.824298    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.824567 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.824494    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.824612 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.824588    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.927240 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.926890    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.927799 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.927768    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.927989 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.927869    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.927989 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.927917    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.928137 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.928112    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drnth\" (UniqueName: \"kubernetes.io/projected/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-kube-api-access-drnth\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.928266 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:14:14.928192    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-serving-cert: secret "maas-api-e2e-shared-a-w4-2cddf2-serving-cert" not found
Nov 02 14:14:14.928352 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:14:14.928280    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-maas-api-tls podName:037046d9-f13b-4a27-a9d2-6b113f1a9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:14:15.428254481 +0000 UTC m=+1787.387630083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-maas-api-tls") pod "maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk" (UID: "037046d9-f13b-4a27-a9d2-6b113f1a9fc6") : secret "maas-api-e2e-shared-a-w4-2cddf2-serving-cert" not found
Nov 02 14:14:14.931237 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.931190    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:14.950461 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:14.950429    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drnth\" (UniqueName: \"kubernetes.io/projected/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-kube-api-access-drnth\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:15.433675 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:15.433634    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:15.436295 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:15.436259    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:15.594689 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:15.594652    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:15.920927 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:15.920882    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"]
Nov 02 14:14:15.922171 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:14:15.922133    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037046d9_f13b_4a27_a9d2_6b113f1a9fc6.slice/crio-df06e69df627967b5f2afcbc650af4a2fd6333a39a19d422b6a2920a2c7048b5 WatchSource:0}: Error finding container df06e69df627967b5f2afcbc650af4a2fd6333a39a19d422b6a2920a2c7048b5: Status 404 returned error can't find the container with id df06e69df627967b5f2afcbc650af4a2fd6333a39a19d422b6a2920a2c7048b5
Nov 02 14:14:16.728949 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.728848    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"]
Nov 02 14:14:16.733068 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.733043    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.764846 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.764812    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"]
Nov 02 14:14:16.865308 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.865270    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzpcx\" (UniqueName: \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-kube-api-access-nzpcx\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.865308 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.865320    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.865612 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.865416    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.865612 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.865467    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.865612 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.865533    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.865752 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.865639    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.865752 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.865676    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.865752 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.865712    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.865907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.865892    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d196b8-eeb0-4933-b320-2682b74cb13c-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.892669 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.892630    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk" event={"ID":"037046d9-f13b-4a27-a9d2-6b113f1a9fc6","Type":"ContainerStarted","Data":"1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3"}
Nov 02 14:14:16.892851 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.892692    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk" event={"ID":"037046d9-f13b-4a27-a9d2-6b113f1a9fc6","Type":"ContainerStarted","Data":"df06e69df627967b5f2afcbc650af4a2fd6333a39a19d422b6a2920a2c7048b5"}
Nov 02 14:14:16.892851 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.892828    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:16.933762 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.933694    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk" podStartSLOduration=2.520506329 podStartE2EDuration="2.933674091s" totalImagesPullingTime="413.167762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:16.931450494 +0000 UTC m=+1788.890826101" watchObservedRunningTime="2026-11-02 14:14:16.933674091 +0000 UTC m=+1788.893049699"
Nov 02 14:14:16.967567 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.967514    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.967785 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.967580    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.967785 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.967615    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.967785 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.967672    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d196b8-eeb0-4933-b320-2682b74cb13c-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.967785 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.967744    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpcx\" (UniqueName: \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-kube-api-access-nzpcx\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.967785 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.967774    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.968183 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.967810    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.968183 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.967842    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.968183 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.967869    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.968338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.968224    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.968572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.968536    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.968658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.968555    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.968742 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.968717    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d196b8-eeb0-4933-b320-2682b74cb13c-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.968801 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.968734    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.970300 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.970275    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.970502 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.970484    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.989346 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.989309    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:16.989525 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:16.989480    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpcx\" (UniqueName: \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-kube-api-access-nzpcx\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:17.050661 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:17.050625    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:17.393020 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:17.392911    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"]
Nov 02 14:14:17.396224 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:14:17.396181    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d196b8_eeb0_4933_b320_2682b74cb13c.slice/crio-a2bd3c5e04f52adcd075f54893c9bae2b66229441a9a953744218916fe69cdfb WatchSource:0}: Error finding container a2bd3c5e04f52adcd075f54893c9bae2b66229441a9a953744218916fe69cdfb: Status 404 returned error can't find the container with id a2bd3c5e04f52adcd075f54893c9bae2b66229441a9a953744218916fe69cdfb
Nov 02 14:14:17.398970 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:17.398921    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:17.399100 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:17.399001    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:17.399100 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:17.399031    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:17.912075 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:17.912026    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" event={"ID":"e3d196b8-eeb0-4933-b320-2682b74cb13c","Type":"ContainerStarted","Data":"069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c"}
Nov 02 14:14:17.912516 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:17.912109    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" event={"ID":"e3d196b8-eeb0-4933-b320-2682b74cb13c","Type":"ContainerStarted","Data":"a2bd3c5e04f52adcd075f54893c9bae2b66229441a9a953744218916fe69cdfb"}
Nov 02 14:14:18.058063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:18.058020    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:18.064125 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:18.064084    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 14:14:18.064439 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:18.064403    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 14:14:19.068961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:19.068767    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 14:14:19.068961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:19.068829    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 14:14:20.053576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:20.053529    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 14:14:20.053894 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:20.053865    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 14:14:21.066597 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:21.066380    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 14:14:21.067159 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:21.067123    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 14:14:22.075783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.075743    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 14:14:22.076464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.076423    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 14:14:22.099626 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.092403    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" podStartSLOduration=6.092377411 podStartE2EDuration="6.092377411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:17.957441869 +0000 UTC m=+1789.916817490" watchObservedRunningTime="2026-11-02 14:14:22.092377411 +0000 UTC m=+1794.051753020"
Nov 02 14:14:22.101272 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.101237    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"]
Nov 02 14:14:22.113017 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.112989    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.120180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.120141    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"]
Nov 02 14:14:22.137104 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.137074    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.137243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.137135    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.137243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.137190    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.137243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.137208    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.137455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.137271    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.137455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.137312    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.137455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.137329    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.137455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.137347    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fsll\" (UniqueName: \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-kube-api-access-9fsll\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.137455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.137414    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982cc2a7-4243-406e-be09-462c0dd2d06a-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.238689 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.238642    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.238890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.238712    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.238890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.238733    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.238890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.238794    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.238890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.238844    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.238890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.238861    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.238890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.238886    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fsll\" (UniqueName: \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-kube-api-access-9fsll\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.239264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.238930    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982cc2a7-4243-406e-be09-462c0dd2d06a-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.239264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.239013    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.239264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.239146    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.239264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.239181    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.239480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.239398    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.239698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.239668    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.239822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.239672    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982cc2a7-4243-406e-be09-462c0dd2d06a-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.241475 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.241447    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.241681 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.241662    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.275021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.274856    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fsll\" (UniqueName: \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-kube-api-access-9fsll\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.275871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.275804    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.439791 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.439692    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:22.795470 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.795435    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"]
Nov 02 14:14:22.797983 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:14:22.797925    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982cc2a7_4243_406e_be09_462c0dd2d06a.slice/crio-5233af024d98ad7b0286b8e84da34d965fdb2a48cc7d5c612a20d1fd080ddf20 WatchSource:0}: Error finding container 5233af024d98ad7b0286b8e84da34d965fdb2a48cc7d5c612a20d1fd080ddf20: Status 404 returned error can't find the container with id 5233af024d98ad7b0286b8e84da34d965fdb2a48cc7d5c612a20d1fd080ddf20
Nov 02 14:14:22.800848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.800812    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:22.800976 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.800887    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:22.800976 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.800924    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:22.918353 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:22.917616    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:14:23.010097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.009918    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" event={"ID":"982cc2a7-4243-406e-be09-462c0dd2d06a","Type":"ContainerStarted","Data":"f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d"}
Nov 02 14:14:23.010097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.010062    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" event={"ID":"982cc2a7-4243-406e-be09-462c0dd2d06a","Type":"ContainerStarted","Data":"5233af024d98ad7b0286b8e84da34d965fdb2a48cc7d5c612a20d1fd080ddf20"}
Nov 02 14:14:23.091024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.071588    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podStartSLOduration=1.071567475 podStartE2EDuration="1.071567475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:23.070822518 +0000 UTC m=+1795.030198150" watchObservedRunningTime="2026-11-02 14:14:23.071567475 +0000 UTC m=+1795.030943084"
Nov 02 14:14:23.092013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.077550    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 14:14:23.092313 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.092278    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 14:14:23.440378 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.440286    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:23.442361 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.442333    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:14:23.442496 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.442399    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:14:23.657162 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.657123    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"]
Nov 02 14:14:23.661506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.661477    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.677684 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.677651    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-4b7759-serving-cert\""
Nov 02 14:14:23.717168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.717071    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"]
Nov 02 14:14:23.733376 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.733335    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"]
Nov 02 14:14:23.739144 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.739103    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.771021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.768549    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"]
Nov 02 14:14:23.785490 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.785451    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.785490 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.785501    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rs7n\" (UniqueName: \"kubernetes.io/projected/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-kube-api-access-7rs7n\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.785754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.785654    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.785876 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.785856    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.886765    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rs7n\" (UniqueName: \"kubernetes.io/projected/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-kube-api-access-7rs7n\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.886830    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.886871    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.886899    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.886924    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nckt\" (UniqueName: \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-kube-api-access-8nckt\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.887013    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.887045    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.887094    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.887152    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58cd9dee-8dfb-429e-a35b-594a1324f077-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.887271    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.887306    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.887340    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.887472    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.891959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.891056    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.894418 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.894391    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.894565 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.894447    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.926911 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.926875    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rs7n\" (UniqueName: \"kubernetes.io/projected/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-kube-api-access-7rs7n\") pod \"maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.983153 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.982086    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:23.990636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.989363    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58cd9dee-8dfb-429e-a35b-594a1324f077-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.990636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.989539    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.990636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.989570    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.990636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.989730    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.990636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.989788    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.990636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.990563    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.990636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.990599    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nckt\" (UniqueName: \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-kube-api-access-8nckt\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.991039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.990840    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.991039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.990907    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:23.991356 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:23.991268    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:24.009005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.004619    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:24.009005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.004854    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58cd9dee-8dfb-429e-a35b-594a1324f077-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:24.009005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.005425    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:24.009005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.005571    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:24.009005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.006193    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:24.011007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.010874    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:24.026405 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.026376    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:24.026553 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.026460    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nckt\" (UniqueName: \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-kube-api-access-8nckt\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:24.070957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.068230    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:24.073329 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.072797    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 02 14:14:24.073329 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.072875    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 02 14:14:24.382491 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.382450    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"]
Nov 02 14:14:24.383636 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:14:24.383603    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25add76_ab8b_4fee_8a5d_484fc0ce58ff.slice/crio-f81a33bc5dd04601521a4f7bd2ec0246e87d4ebf9043d55704be2bd1e5eb0d91 WatchSource:0}: Error finding container f81a33bc5dd04601521a4f7bd2ec0246e87d4ebf9043d55704be2bd1e5eb0d91: Status 404 returned error can't find the container with id f81a33bc5dd04601521a4f7bd2ec0246e87d4ebf9043d55704be2bd1e5eb0d91
Nov 02 14:14:24.440366 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.440328    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:14:24.440576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.440392    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:14:24.480644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.480595    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"]
Nov 02 14:14:24.484609 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:14:24.484579    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58cd9dee_8dfb_429e_a35b_594a1324f077.slice/crio-226f70e5095d3ca6b4da43a8bb49315f03892261f6304d1a9d5aaa121f508f17 WatchSource:0}: Error finding container 226f70e5095d3ca6b4da43a8bb49315f03892261f6304d1a9d5aaa121f508f17: Status 404 returned error can't find the container with id 226f70e5095d3ca6b4da43a8bb49315f03892261f6304d1a9d5aaa121f508f17
Nov 02 14:14:24.488244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.488208    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:24.488338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.488285    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:24.488338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:24.488315    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:14:25.037855 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.037754    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42" event={"ID":"d25add76-ab8b-4fee-8a5d-484fc0ce58ff","Type":"ContainerStarted","Data":"f81a33bc5dd04601521a4f7bd2ec0246e87d4ebf9043d55704be2bd1e5eb0d91"}
Nov 02 14:14:25.043830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.043697    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" event={"ID":"58cd9dee-8dfb-429e-a35b-594a1324f077","Type":"ContainerStarted","Data":"b11532b3af72427c088e7858abb055a3ef6f4b7e0e5749b0f6c287e6d5fc27b1"}
Nov 02 14:14:25.043830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.043762    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" event={"ID":"58cd9dee-8dfb-429e-a35b-594a1324f077","Type":"ContainerStarted","Data":"226f70e5095d3ca6b4da43a8bb49315f03892261f6304d1a9d5aaa121f508f17"}
Nov 02 14:14:25.079140 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.078693    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:25.079140 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.078822    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 14:14:25.079140 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.078867    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 14:14:25.105080 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.097716    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podStartSLOduration=2.097697075 podStartE2EDuration="2.097697075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:25.0963674 +0000 UTC m=+1797.055743008" watchObservedRunningTime="2026-11-02 14:14:25.097697075 +0000 UTC m=+1797.057072682"
Nov 02 14:14:25.441574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.441459    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:14:25.441574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.441529    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:14:25.607563 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.607525    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:25.609193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.609174    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:25.617110 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:25.617073    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:14:26.089580 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:26.089093    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 14:14:26.089580 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:26.089161    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 14:14:26.130674 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:26.127112    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42" event={"ID":"d25add76-ab8b-4fee-8a5d-484fc0ce58ff","Type":"ContainerStarted","Data":"071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7"}
Nov 02 14:14:26.130674 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:26.127820    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:26.225113 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:26.225035    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42" podStartSLOduration=2.707743345 podStartE2EDuration="3.225012373s" totalImagesPullingTime="517.269028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:26.195644137 +0000 UTC m=+1798.155019748" watchObservedRunningTime="2026-11-02 14:14:26.225012373 +0000 UTC m=+1798.184387981"
Nov 02 14:14:26.440736 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:26.440627    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:14:26.440736 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:26.440687    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:14:27.076960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:27.073419    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 14:14:27.076960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:27.073506    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 14:14:27.442023 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:27.440439    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:14:27.442023 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:27.440522    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:14:28.081981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:28.081457    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 14:14:28.081981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:28.081547    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 14:14:28.522960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:28.522518    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:14:28.522960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:28.522594    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:14:28.702465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:28.702422    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:14:28.725040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:28.722208    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:14:29.075713 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:29.075670    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 14:14:29.076058 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:29.076017    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 14:14:29.441026 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:29.440353    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:14:29.441026 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:29.440415    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:14:30.075847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:30.075352    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 14:14:30.075847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:30.075454    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 14:14:30.440412 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:30.440296    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:14:30.440412 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:30.440365    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:14:31.072330 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:31.072107    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"]
Nov 02 14:14:31.092962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:31.081313    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy" containerID="cri-o://6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06" gracePeriod=30
Nov 02 14:14:31.102191 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:31.093785    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 14:14:31.102458 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:31.102426    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 14:14:31.455025 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:31.453155    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 02 14:14:31.455025 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:31.453219    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 02 14:14:32.073531 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:32.073366    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 14:14:32.073531 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:32.073435    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 14:14:32.546959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:32.546076    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:32.548714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:32.547239    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:32.548714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:32.548583    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:14:33.077549 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:33.077515    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:33.078105 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:33.077909    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:33.078716 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:33.078694    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:33.153379 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:33.153343    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:14:36.405749 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.405674    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 14:14:36.571805 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.571771    6284 generic.go:356] "Generic (PLEG): container finished" podID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerID="6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06" exitCode=0
Nov 02 14:14:36.571805 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.571861    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" event={"ID":"844a58bb-ce76-46ff-ab23-d86b373c61fb","Type":"ContainerDied","Data":"6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06"}
Nov 02 14:14:36.573062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.571905    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q" event={"ID":"844a58bb-ce76-46ff-ab23-d86b373c61fb","Type":"ContainerDied","Data":"51b11fbb82c23f7debd9852cedcb265b97b61f96132cb1fe6fcf020a9dfda41b"}
Nov 02 14:14:36.573062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.571947    6284 scope.go:122] "RemoveContainer" containerID="6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06"
Nov 02 14:14:36.573062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.572198    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"
Nov 02 14:14:36.573757 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.573727    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844a58bb-ce76-46ff-ab23-d86b373c61fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844a58bb-ce76-46ff-ab23-d86b373c61fb-istiod-ca-cert\") pod \"844a58bb-ce76-46ff-ab23-d86b373c61fb\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") "
Nov 02 14:14:36.573975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.573893    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-data\") pod \"844a58bb-ce76-46ff-ab23-d86b373c61fb\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") "
Nov 02 14:14:36.574115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574095    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-token\" (UniqueName: \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-token\") pod \"844a58bb-ce76-46ff-ab23-d86b373c61fb\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") "
Nov 02 14:14:36.574182 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574143    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-certs\") pod \"844a58bb-ce76-46ff-ab23-d86b373c61fb\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") "
Nov 02 14:14:36.574242 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574181    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-envoy\") pod \"844a58bb-ce76-46ff-ab23-d86b373c61fb\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") "
Nov 02 14:14:36.574242 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574213    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844a58bb-ce76-46ff-ab23-d86b373c61fb-istiod-ca-cert" pod "844a58bb-ce76-46ff-ab23-d86b373c61fb" (UID: "844a58bb-ce76-46ff-ab23-d86b373c61fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:36.574354 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574235    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-podinfo\") pod \"844a58bb-ce76-46ff-ab23-d86b373c61fb\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") "
Nov 02 14:14:36.574354 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574295    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-kube-api-access-8cbl9\" (UniqueName: \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-kube-api-access-8cbl9\") pod \"844a58bb-ce76-46ff-ab23-d86b373c61fb\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") "
Nov 02 14:14:36.574459 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574372    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-credential-socket\") pod \"844a58bb-ce76-46ff-ab23-d86b373c61fb\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") "
Nov 02 14:14:36.574536 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574482    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-data" pod "844a58bb-ce76-46ff-ab23-d86b373c61fb" (UID: "844a58bb-ce76-46ff-ab23-d86b373c61fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:36.574602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574557    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-certs" pod "844a58bb-ce76-46ff-ab23-d86b373c61fb" (UID: "844a58bb-ce76-46ff-ab23-d86b373c61fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:36.574749 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574726    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-socket\") pod \"844a58bb-ce76-46ff-ab23-d86b373c61fb\" (UID: \"844a58bb-ce76-46ff-ab23-d86b373c61fb\") "
Nov 02 14:14:36.574842 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.574823    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-credential-socket" pod "844a58bb-ce76-46ff-ab23-d86b373c61fb" (UID: "844a58bb-ce76-46ff-ab23-d86b373c61fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:36.575146 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.575110    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-socket" pod "844a58bb-ce76-46ff-ab23-d86b373c61fb" (UID: "844a58bb-ce76-46ff-ab23-d86b373c61fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:36.576311 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.575625    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:36.576311 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.575658    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:36.576311 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.575676    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/844a58bb-ce76-46ff-ab23-d86b373c61fb-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:36.576311 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.575692    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:36.576311 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.575708    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:36.576930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.576896    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-envoy" pod "844a58bb-ce76-46ff-ab23-d86b373c61fb" (UID: "844a58bb-ce76-46ff-ab23-d86b373c61fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:36.577179 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.577154    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-token" pod "844a58bb-ce76-46ff-ab23-d86b373c61fb" (UID: "844a58bb-ce76-46ff-ab23-d86b373c61fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:36.577349 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.577327    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-podinfo" pod "844a58bb-ce76-46ff-ab23-d86b373c61fb" (UID: "844a58bb-ce76-46ff-ab23-d86b373c61fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:36.578157 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.578132    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-kube-api-access-8cbl9" pod "844a58bb-ce76-46ff-ab23-d86b373c61fb" (UID: "844a58bb-ce76-46ff-ab23-d86b373c61fb"). InnerVolumeSpecName "kube-api-access-8cbl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:36.604584 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.604558    6284 scope.go:122] "RemoveContainer" containerID="6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06"
Nov 02 14:14:36.604995 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:14:36.604963    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06\": container with ID starting with 6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06 not found: ID does not exist" containerID="6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06"
Nov 02 14:14:36.605070 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.605010    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06"} err="failed to get container status \"6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06\": rpc error: code = NotFound desc = could not find container \"6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06\": container with ID starting with 6fd2ce8ca6a8c2a4c718f75040b6c7e49fa957c2748f08502cb7c632b70b2f06 not found: ID does not exist"
Nov 02 14:14:36.677007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.676879    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:36.677007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.676913    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cbl9\" (UniqueName: \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-kube-api-access-8cbl9\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:36.677007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.676930    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:36.677007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.676957    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/844a58bb-ce76-46ff-ab23-d86b373c61fb-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:36.914592 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.914554    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"]
Nov 02 14:14:36.923005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:36.922957    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-sdm2q"]
Nov 02 14:14:38.173795 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:38.173755    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" path="/var/lib/kubelet/pods/844a58bb-ce76-46ff-ab23-d86b373c61fb/volumes"
Nov 02 14:14:43.512979 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:43.512919    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"]
Nov 02 14:14:43.513543 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:43.513214    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy" containerID="cri-o://b11532b3af72427c088e7858abb055a3ef6f4b7e0e5749b0f6c287e6d5fc27b1" gracePeriod=30
Nov 02 14:14:43.523309 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:43.523270    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"]
Nov 02 14:14:43.523581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:43.523538    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy" containerID="cri-o://7f3bebce1cd32d74025c7ade5894d2635c8853fe6fbaf6f6ae3bd26d55a65bb5" gracePeriod=30
Nov 02 14:14:43.897436 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:14:43.897324    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:14:48.015688 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:48.015639    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:14:48.016177 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:48.015700    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:14:48.080022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:48.079979    6284 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 14:14:48.080277 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:48.080053    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:14:48.685717 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:48.685521    6284 generic.go:356] "Generic (PLEG): container finished" podID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerID="7f3bebce1cd32d74025c7ade5894d2635c8853fe6fbaf6f6ae3bd26d55a65bb5" exitCode=0
Nov 02 14:14:48.685717 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:48.685644    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" event={"ID":"6ca316e7-a65a-4964-a3f9-73b192cb285f","Type":"ContainerDied","Data":"7f3bebce1cd32d74025c7ade5894d2635c8853fe6fbaf6f6ae3bd26d55a65bb5"}
Nov 02 14:14:48.688367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:48.688330    6284 generic.go:356] "Generic (PLEG): container finished" podID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerID="b11532b3af72427c088e7858abb055a3ef6f4b7e0e5749b0f6c287e6d5fc27b1" exitCode=0
Nov 02 14:14:48.688529 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:48.688429    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" event={"ID":"58cd9dee-8dfb-429e-a35b-594a1324f077","Type":"ContainerDied","Data":"b11532b3af72427c088e7858abb055a3ef6f4b7e0e5749b0f6c287e6d5fc27b1"}
Nov 02 14:14:48.965964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:48.963857    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:49.003508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.003473    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:14:49.023557 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.023526    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58cd9dee-8dfb-429e-a35b-594a1324f077-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58cd9dee-8dfb-429e-a35b-594a1324f077-istiod-ca-cert\") pod \"58cd9dee-8dfb-429e-a35b-594a1324f077\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") "
Nov 02 14:14:49.024097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.023568    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-envoy\") pod \"58cd9dee-8dfb-429e-a35b-594a1324f077\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") "
Nov 02 14:14:49.024097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.023989    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58cd9dee-8dfb-429e-a35b-594a1324f077-istiod-ca-cert" pod "58cd9dee-8dfb-429e-a35b-594a1324f077" (UID: "58cd9dee-8dfb-429e-a35b-594a1324f077"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:49.026367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.026332    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-envoy" pod "58cd9dee-8dfb-429e-a35b-594a1324f077" (UID: "58cd9dee-8dfb-429e-a35b-594a1324f077"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.026547 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.026522    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-credential-socket\") pod \"58cd9dee-8dfb-429e-a35b-594a1324f077\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") "
Nov 02 14:14:49.026617 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.026564    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-data\") pod \"58cd9dee-8dfb-429e-a35b-594a1324f077\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") "
Nov 02 14:14:49.026791 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.026773    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-certs\") pod \"58cd9dee-8dfb-429e-a35b-594a1324f077\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") "
Nov 02 14:14:49.026889 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.026792    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-credential-socket" pod "58cd9dee-8dfb-429e-a35b-594a1324f077" (UID: "58cd9dee-8dfb-429e-a35b-594a1324f077"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.026889 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.026807    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-podinfo\") pod \"58cd9dee-8dfb-429e-a35b-594a1324f077\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") "
Nov 02 14:14:49.026889 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.026834    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-data" pod "58cd9dee-8dfb-429e-a35b-594a1324f077" (UID: "58cd9dee-8dfb-429e-a35b-594a1324f077"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.026889 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.026855    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-kube-api-access-8nckt\" (UniqueName: \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-kube-api-access-8nckt\") pod \"58cd9dee-8dfb-429e-a35b-594a1324f077\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") "
Nov 02 14:14:49.027132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.026969    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-socket\") pod \"58cd9dee-8dfb-429e-a35b-594a1324f077\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") "
Nov 02 14:14:49.027132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.026997    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-token\" (UniqueName: \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-token\") pod \"58cd9dee-8dfb-429e-a35b-594a1324f077\" (UID: \"58cd9dee-8dfb-429e-a35b-594a1324f077\") "
Nov 02 14:14:49.027132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.027003    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-certs" pod "58cd9dee-8dfb-429e-a35b-594a1324f077" (UID: "58cd9dee-8dfb-429e-a35b-594a1324f077"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.027344 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.027208    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-socket" pod "58cd9dee-8dfb-429e-a35b-594a1324f077" (UID: "58cd9dee-8dfb-429e-a35b-594a1324f077"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.027649 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.027606    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.027649 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.027649    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.027837 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.027664    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58cd9dee-8dfb-429e-a35b-594a1324f077-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.027837 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.027678    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.027837 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.027692    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.027837 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.027708    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.031208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.031161    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-kube-api-access-8nckt" pod "58cd9dee-8dfb-429e-a35b-594a1324f077" (UID: "58cd9dee-8dfb-429e-a35b-594a1324f077"). InnerVolumeSpecName "kube-api-access-8nckt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:49.031546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.031521    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-token" pod "58cd9dee-8dfb-429e-a35b-594a1324f077" (UID: "58cd9dee-8dfb-429e-a35b-594a1324f077"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:49.031708 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.031692    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-podinfo" pod "58cd9dee-8dfb-429e-a35b-594a1324f077" (UID: "58cd9dee-8dfb-429e-a35b-594a1324f077"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:49.134983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.133955    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca316e7-a65a-4964-a3f9-73b192cb285f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca316e7-a65a-4964-a3f9-73b192cb285f-istiod-ca-cert\") pod \"6ca316e7-a65a-4964-a3f9-73b192cb285f\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") "
Nov 02 14:14:49.134983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.134006    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-podinfo\") pod \"6ca316e7-a65a-4964-a3f9-73b192cb285f\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") "
Nov 02 14:14:49.134983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.134050    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-data\") pod \"6ca316e7-a65a-4964-a3f9-73b192cb285f\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") "
Nov 02 14:14:49.134983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.134195    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-credential-socket\") pod \"6ca316e7-a65a-4964-a3f9-73b192cb285f\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") "
Nov 02 14:14:49.134983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.134268    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-envoy\") pod \"6ca316e7-a65a-4964-a3f9-73b192cb285f\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") "
Nov 02 14:14:49.134983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.134300    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-token\" (UniqueName: \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-token\") pod \"6ca316e7-a65a-4964-a3f9-73b192cb285f\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") "
Nov 02 14:14:49.134983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.134368    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-certs\") pod \"6ca316e7-a65a-4964-a3f9-73b192cb285f\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") "
Nov 02 14:14:49.134983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.134401    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-socket\") pod \"6ca316e7-a65a-4964-a3f9-73b192cb285f\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") "
Nov 02 14:14:49.134983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.134445    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-kube-api-access-mnpxf\" (UniqueName: \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-kube-api-access-mnpxf\") pod \"6ca316e7-a65a-4964-a3f9-73b192cb285f\" (UID: \"6ca316e7-a65a-4964-a3f9-73b192cb285f\") "
Nov 02 14:14:49.142739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.134999    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.142739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.135023    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nckt\" (UniqueName: \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-kube-api-access-8nckt\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.142739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.135036    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58cd9dee-8dfb-429e-a35b-594a1324f077-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.142739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.135669    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-credential-socket" pod "6ca316e7-a65a-4964-a3f9-73b192cb285f" (UID: "6ca316e7-a65a-4964-a3f9-73b192cb285f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.142739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.135991    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca316e7-a65a-4964-a3f9-73b192cb285f-istiod-ca-cert" pod "6ca316e7-a65a-4964-a3f9-73b192cb285f" (UID: "6ca316e7-a65a-4964-a3f9-73b192cb285f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:49.142739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.136639    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-certs" pod "6ca316e7-a65a-4964-a3f9-73b192cb285f" (UID: "6ca316e7-a65a-4964-a3f9-73b192cb285f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.142739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.136825    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-data" pod "6ca316e7-a65a-4964-a3f9-73b192cb285f" (UID: "6ca316e7-a65a-4964-a3f9-73b192cb285f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.142739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.137081    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-socket" pod "6ca316e7-a65a-4964-a3f9-73b192cb285f" (UID: "6ca316e7-a65a-4964-a3f9-73b192cb285f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.152207 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.152161    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-kube-api-access-mnpxf" pod "6ca316e7-a65a-4964-a3f9-73b192cb285f" (UID: "6ca316e7-a65a-4964-a3f9-73b192cb285f"). InnerVolumeSpecName "kube-api-access-mnpxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:49.153279 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.153191    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-token" pod "6ca316e7-a65a-4964-a3f9-73b192cb285f" (UID: "6ca316e7-a65a-4964-a3f9-73b192cb285f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:49.153718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.153659    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-podinfo" pod "6ca316e7-a65a-4964-a3f9-73b192cb285f" (UID: "6ca316e7-a65a-4964-a3f9-73b192cb285f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:49.159022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.157195    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-envoy" pod "6ca316e7-a65a-4964-a3f9-73b192cb285f" (UID: "6ca316e7-a65a-4964-a3f9-73b192cb285f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.237016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.236901    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnpxf\" (UniqueName: \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-kube-api-access-mnpxf\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.237016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.237006    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca316e7-a65a-4964-a3f9-73b192cb285f-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.237016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.237019    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.237338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.237032    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.237338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.237046    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.237338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.237059    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.237338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.237072    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca316e7-a65a-4964-a3f9-73b192cb285f-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.237338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.237085    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.237338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.237097    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca316e7-a65a-4964-a3f9-73b192cb285f-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.701335 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.701302    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"
Nov 02 14:14:49.702036 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.701303    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl" event={"ID":"6ca316e7-a65a-4964-a3f9-73b192cb285f","Type":"ContainerDied","Data":"98ac53be24ee0519e64a46f982fc832d7dd28444cfb92578601edcb2b1e039b6"}
Nov 02 14:14:49.702566 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.702536    6284 scope.go:122] "RemoveContainer" containerID="7f3bebce1cd32d74025c7ade5894d2635c8853fe6fbaf6f6ae3bd26d55a65bb5"
Nov 02 14:14:49.704268 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.704241    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6" event={"ID":"58cd9dee-8dfb-429e-a35b-594a1324f077","Type":"ContainerDied","Data":"226f70e5095d3ca6b4da43a8bb49315f03892261f6304d1a9d5aaa121f508f17"}
Nov 02 14:14:49.704389 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.704293    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"
Nov 02 14:14:49.716774 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.716745    6284 scope.go:122] "RemoveContainer" containerID="b11532b3af72427c088e7858abb055a3ef6f4b7e0e5749b0f6c287e6d5fc27b1"
Nov 02 14:14:49.765552 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.765511    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"]
Nov 02 14:14:49.776777 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.776740    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-ncscl"]
Nov 02 14:14:49.812918 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.812827    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"]
Nov 02 14:14:49.817349 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:49.817314    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-vjwv6"]
Nov 02 14:14:50.172004 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.171901    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" path="/var/lib/kubelet/pods/58cd9dee-8dfb-429e-a35b-594a1324f077/volumes"
Nov 02 14:14:50.172500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.172426    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" path="/var/lib/kubelet/pods/6ca316e7-a65a-4964-a3f9-73b192cb285f/volumes"
Nov 02 14:14:50.264663 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.264625    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"]
Nov 02 14:14:50.265362 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.265329    6284 cpu_manager.go:414] "containerMap: removing container" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy"
Nov 02 14:14:50.265362 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.265356    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy"
Nov 02 14:14:50.265616 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.265377    6284 cpu_manager.go:414] "containerMap: removing container" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy"
Nov 02 14:14:50.265616 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.265382    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy"
Nov 02 14:14:50.265616 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.265427    6284 cpu_manager.go:414] "containerMap: removing container" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy"
Nov 02 14:14:50.265616 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.265437    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy"
Nov 02 14:14:50.265616 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.265580    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca316e7-a65a-4964-a3f9-73b192cb285f" containerName="istio-proxy"
Nov 02 14:14:50.265616 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.265591    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="58cd9dee-8dfb-429e-a35b-594a1324f077" containerName="istio-proxy"
Nov 02 14:14:50.265616 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.265602    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="844a58bb-ce76-46ff-ab23-d86b373c61fb" containerName="istio-proxy"
Nov 02 14:14:50.270902 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.270872    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.295540 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.295503    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"]
Nov 02 14:14:50.298694 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.298653    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-21baa7-serving-cert\""
Nov 02 14:14:50.354502 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.354468    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.354719 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.354545    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3779b6eb-7714-44e3-a3ae-75e381312aed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.354817 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.354790    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.354885 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.354864    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbjk\" (UniqueName: \"kubernetes.io/projected/3779b6eb-7714-44e3-a3ae-75e381312aed-kube-api-access-drbjk\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.456669 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.456567    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbjk\" (UniqueName: \"kubernetes.io/projected/3779b6eb-7714-44e3-a3ae-75e381312aed-kube-api-access-drbjk\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.456871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.456673    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.456871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.456708    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3779b6eb-7714-44e3-a3ae-75e381312aed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.456871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.456839    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.457723 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.457695    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3779b6eb-7714-44e3-a3ae-75e381312aed-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.459609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.459585    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.459793 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.459628    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.478444 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.478414    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbjk\" (UniqueName: \"kubernetes.io/projected/3779b6eb-7714-44e3-a3ae-75e381312aed-kube-api-access-drbjk\") pod \"maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.588036 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.585628    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:50.910253 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:50.909848    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"]
Nov 02 14:14:50.913550 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:14:50.913505    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3779b6eb_7714_44e3_a3ae_75e381312aed.slice/crio-38dd79117055950cc9226a0fc442ed33c818ae7713d177211dca380e249eb300 WatchSource:0}: Error finding container 38dd79117055950cc9226a0fc442ed33c818ae7713d177211dca380e249eb300: Status 404 returned error can't find the container with id 38dd79117055950cc9226a0fc442ed33c818ae7713d177211dca380e249eb300
Nov 02 14:14:51.735263 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:51.735218    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv" event={"ID":"3779b6eb-7714-44e3-a3ae-75e381312aed","Type":"ContainerStarted","Data":"c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407"}
Nov 02 14:14:51.735784 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:51.735341    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv" event={"ID":"3779b6eb-7714-44e3-a3ae-75e381312aed","Type":"ContainerStarted","Data":"38dd79117055950cc9226a0fc442ed33c818ae7713d177211dca380e249eb300"}
Nov 02 14:14:51.735784 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:51.735635    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:57.748329 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:57.748301    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:14:57.786396 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:14:57.786341    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv" podStartSLOduration=7.329411666 podStartE2EDuration="7.786324736s" totalImagesPullingTime="456.91307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:51.777351585 +0000 UTC m=+1823.736727193" watchObservedRunningTime="2026-11-02 14:14:57.786324736 +0000 UTC m=+1829.745700344"
Nov 02 14:15:03.282503 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.282469    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"]
Nov 02 14:15:03.286797 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.286765    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.303138 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.303109    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-4c2811-serving-cert\""
Nov 02 14:15:03.319129 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.319075    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"]
Nov 02 14:15:03.350276 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.350237    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdkmd\" (UniqueName: \"kubernetes.io/projected/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-kube-api-access-mdkmd\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.350465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.350381    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.350465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.350456    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.350605 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.350592    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.451776 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.451729    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.452036 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.451814    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdkmd\" (UniqueName: \"kubernetes.io/projected/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-kube-api-access-mdkmd\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.452036 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.451868    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.452036 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.451901    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.452036 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:15:03.451898    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-serving-cert: secret "maas-api-e2e-shared-b-w4-4c2811-serving-cert" not found
Nov 02 14:15:03.452275 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:15:03.452103    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-maas-api-tls podName:388bb99e-a56a-479d-964c-0e1dfaa5ec3a nodeName:}" failed. No retries permitted until 2026-11-02 14:15:03.952079385 +0000 UTC m=+1835.911454984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-maas-api-tls") pod "maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9" (UID: "388bb99e-a56a-479d-964c-0e1dfaa5ec3a") : secret "maas-api-e2e-shared-b-w4-4c2811-serving-cert" not found
Nov 02 14:15:03.452651 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.452627    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.454745 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.454713    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.474019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.473982    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdkmd\" (UniqueName: \"kubernetes.io/projected/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-kube-api-access-mdkmd\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.959000 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.958929    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:03.961666 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:03.961641    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:04.200794 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:04.200749    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:04.527152 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:04.527114    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"]
Nov 02 14:15:04.835435 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:04.835252    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9" event={"ID":"388bb99e-a56a-479d-964c-0e1dfaa5ec3a","Type":"ContainerStarted","Data":"db21c6ada0806ad6d63269d50f878d6eba2a839fec11c42c74f3fed8b379da23"}
Nov 02 14:15:05.844475 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:05.844431    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9" event={"ID":"388bb99e-a56a-479d-964c-0e1dfaa5ec3a","Type":"ContainerStarted","Data":"e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095"}
Nov 02 14:15:05.844898 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:05.844505    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:05.936084 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:05.935853    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9" podStartSLOduration=2.521752646 podStartE2EDuration="2.935830016s" totalImagesPullingTime="414.07737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:05.933975433 +0000 UTC m=+1837.893351040" watchObservedRunningTime="2026-11-02 14:15:05.935830016 +0000 UTC m=+1837.895205624"
Nov 02 14:15:11.857728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:11.857693    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:15:13.897416 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:15:13.897378    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:15:14.477881 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.477832    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"]
Nov 02 14:15:14.482278 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.482257    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.494571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.494524    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-fcefcc-serving-cert\""
Nov 02 14:15:14.508420 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.508384    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"]
Nov 02 14:15:14.611555 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.611493    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/025e41b6-cbe6-4e32-9b90-57d466d13c22-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.611555 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.611555    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.611833 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.611805    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7g4\" (UniqueName: \"kubernetes.io/projected/025e41b6-cbe6-4e32-9b90-57d466d13c22-kube-api-access-db7g4\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.611876 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.611855    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.713116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.713074    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7g4\" (UniqueName: \"kubernetes.io/projected/025e41b6-cbe6-4e32-9b90-57d466d13c22-kube-api-access-db7g4\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.713336 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.713126    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.713336 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:15:14.713265    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-serving-cert: secret "maas-api-e2e-shared-b-w2-fcefcc-serving-cert" not found
Nov 02 14:15:14.713336 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.713285    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/025e41b6-cbe6-4e32-9b90-57d466d13c22-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.713336 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.713318    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.713336 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:15:14.713338    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-maas-api-tls podName:025e41b6-cbe6-4e32-9b90-57d466d13c22 nodeName:}" failed. No retries permitted until 2026-11-02 14:15:15.213320559 +0000 UTC m=+1847.172696167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-maas-api-tls") pod "maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94" (UID: "025e41b6-cbe6-4e32-9b90-57d466d13c22") : secret "maas-api-e2e-shared-b-w2-fcefcc-serving-cert" not found
Nov 02 14:15:14.714034 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.714003    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/025e41b6-cbe6-4e32-9b90-57d466d13c22-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.716499 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.716471    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:14.736040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:14.736004    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7g4\" (UniqueName: \"kubernetes.io/projected/025e41b6-cbe6-4e32-9b90-57d466d13c22-kube-api-access-db7g4\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:15.218786 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:15.218685    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:15.221277 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:15.221243    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:15.398780 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:15.398268    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:15.737997 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:15.737692    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"]
Nov 02 14:15:15.749355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:15.749309    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:15:15.910864 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:15.910823    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94" event={"ID":"025e41b6-cbe6-4e32-9b90-57d466d13c22","Type":"ContainerStarted","Data":"eac49772c0c7834ce94ee922ba6f63a4b62163062dd1f3aa5faac2924f12f8f3"}
Nov 02 14:15:16.920470 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:16.920430    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94" event={"ID":"025e41b6-cbe6-4e32-9b90-57d466d13c22","Type":"ContainerStarted","Data":"76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6"}
Nov 02 14:15:16.921040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:16.920508    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:16.962639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:16.962579    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94" podStartSLOduration=2.613405649 podStartE2EDuration="2.96256146s" totalImagesPullingTime="349.155811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:16.959592466 +0000 UTC m=+1848.918968073" watchObservedRunningTime="2026-11-02 14:15:16.96256146 +0000 UTC m=+1848.921937061"
Nov 02 14:15:22.929647 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:22.929606    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94" podUID="025e41b6-cbe6-4e32-9b90-57d466d13c22" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.109:8443/health\": dial tcp 10.134.0.109:8443: connect: connection refused"
Nov 02 14:15:27.037809 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:27.037774    6284 generic.go:356] "Generic (PLEG): container finished" podID="025e41b6-cbe6-4e32-9b90-57d466d13c22" containerID="76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6" exitCode=1
Nov 02 14:15:27.038285 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:27.037842    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94" event={"ID":"025e41b6-cbe6-4e32-9b90-57d466d13c22","Type":"ContainerDied","Data":"76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6"}
Nov 02 14:15:27.038327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:27.038298    6284 scope.go:122] "RemoveContainer" containerID="76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6"
Nov 02 14:15:28.046770 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:28.046735    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94" event={"ID":"025e41b6-cbe6-4e32-9b90-57d466d13c22","Type":"ContainerStarted","Data":"b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f"}
Nov 02 14:15:33.047754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:33.047713    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:33.053065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:33.053036    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:15:35.649897 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.649858    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"]
Nov 02 14:15:35.655009 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.654981    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.674699 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.674668    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fcefcc-openshift-default-dockercfg-r7kkx\""
Nov 02 14:15:35.691607 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.691568    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"]
Nov 02 14:15:35.737061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.737023    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.737276 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.737081    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twpsz\" (UniqueName: \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-kube-api-access-twpsz\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.737276 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.737234    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.737398 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.737301    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.737472 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.737445    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.737601 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.737584    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.737641 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.737621    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.737685 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.737654    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.737773 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.737753    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.838728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.838691    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.838737    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.838757    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.838799    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.838832    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpsz\" (UniqueName: \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-kube-api-access-twpsz\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.838874    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.838919    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839347 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.839050    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839347 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.839158    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839347 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.839262    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839347 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.839280    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839503 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.839359    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.839503 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.839394    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.840108 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.840083    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.841551 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.841525    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.841775 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.841754    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.862708 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.862678    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.862867 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.862828    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpsz\" (UniqueName: \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-kube-api-access-twpsz\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:35.972746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:35.972664    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:36.308358 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:36.308324    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"]
Nov 02 14:15:36.310498 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:15:36.310469    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ca6186_3243_459b_ad2a_7d6584ddf0d2.slice/crio-130d78a063cef022d82d33cba08272d0dd28eeb055c36b6c11ac1bf3b7650788 WatchSource:0}: Error finding container 130d78a063cef022d82d33cba08272d0dd28eeb055c36b6c11ac1bf3b7650788: Status 404 returned error can't find the container with id 130d78a063cef022d82d33cba08272d0dd28eeb055c36b6c11ac1bf3b7650788
Nov 02 14:15:36.313731 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:36.313686    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:15:36.313829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:36.313790    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:15:36.313883 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:36.313847    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:15:37.113008 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:37.112964    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" event={"ID":"e0ca6186-3243-459b-ad2a-7d6584ddf0d2","Type":"ContainerStarted","Data":"58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0"}
Nov 02 14:15:37.113417 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:37.113041    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" event={"ID":"e0ca6186-3243-459b-ad2a-7d6584ddf0d2","Type":"ContainerStarted","Data":"130d78a063cef022d82d33cba08272d0dd28eeb055c36b6c11ac1bf3b7650788"}
Nov 02 14:15:37.181069 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:37.180999    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" podStartSLOduration=2.18097629 podStartE2EDuration="2.18097629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:37.170545447 +0000 UTC m=+1869.129921055" watchObservedRunningTime="2026-11-02 14:15:37.18097629 +0000 UTC m=+1869.140351899"
Nov 02 14:15:37.974958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:37.974402    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:37.995063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:37.993648    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 14:15:37.995063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:37.993708    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 14:15:38.973959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:38.973410    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 14:15:38.973959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:38.973478    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 14:15:39.380668 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.380625    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"]
Nov 02 14:15:39.391383 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.391339    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.394835 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.394803    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"]
Nov 02 14:15:39.414574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.414543    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-2cddf2\"/\"auth-test-model-2cddf2-kserve-self-signed-certs\""
Nov 02 14:15:39.414807 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.414680    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-2cddf2\"/\"default-dockercfg-v4fzf\""
Nov 02 14:15:39.414807 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.414758    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-2cddf2\"/\"openshift-service-ca.crt\""
Nov 02 14:15:39.415013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.414976    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-2cddf2\"/\"kube-root-ca.crt\""
Nov 02 14:15:39.483244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.483204    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-model-cache\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.483864 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.483837    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhmj8\" (UniqueName: \"kubernetes.io/projected/53e527d2-ba69-4c74-8d10-14d8fc00eeed-kube-api-access-lhmj8\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.484190 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.484152    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tls-certs\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.484458 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.484429    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-dshm\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.484693 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.484673    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tmp-dir\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.484876 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.484829    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-home\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.587252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.587062    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tmp-dir\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.587252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.587130    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-home\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.587252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.587177    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-model-cache\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.587252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.587210    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmj8\" (UniqueName: \"kubernetes.io/projected/53e527d2-ba69-4c74-8d10-14d8fc00eeed-kube-api-access-lhmj8\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.587657 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.587315    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tls-certs\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.587657 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.587359    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-dshm\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.589005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.588930    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-home\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.589400 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.589379    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tmp-dir\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.589746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.589728    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-model-cache\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.592996 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.591635    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tls-certs\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.593711 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.593688    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-dshm\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.609801 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.609758    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmj8\" (UniqueName: \"kubernetes.io/projected/53e527d2-ba69-4c74-8d10-14d8fc00eeed-kube-api-access-lhmj8\") pod \"auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.719017 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.717220    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:39.995016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.987085    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 14:15:39.995016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:39.987151    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 14:15:40.184492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.184467    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"]
Nov 02 14:15:40.185279 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:15:40.185249    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53e527d2_ba69_4c74_8d10_14d8fc00eeed.slice/crio-3d7be958a03af014b8f7b8c3c56191247eeb686579aa9274bd87d96792dfb6f1 WatchSource:0}: Error finding container 3d7be958a03af014b8f7b8c3c56191247eeb686579aa9274bd87d96792dfb6f1: Status 404 returned error can't find the container with id 3d7be958a03af014b8f7b8c3c56191247eeb686579aa9274bd87d96792dfb6f1
Nov 02 14:15:40.384363 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.384327    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"]
Nov 02 14:15:40.388531 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.388508    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.415096 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.415052    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"]
Nov 02 14:15:40.493257 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.493222    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4c2811-openshift-default-dockercfg-pqrrz\""
Nov 02 14:15:40.503838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.503791    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.504116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.503858    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.504116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.504032    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.504116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.504084    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.504249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.504179    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.504249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.504214    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.504249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.504240    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpmvw\" (UniqueName: \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-kube-api-access-bpmvw\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.504403 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.504360    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.504461 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.504405    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.605896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.605809    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.605896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.605878    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606189 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.605912    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606189 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.605990    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpmvw\" (UniqueName: \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-kube-api-access-bpmvw\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606189 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.606043    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606189 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.606073    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606397 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.606227    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606397 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.606276    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606397 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.606352    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606397 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.606364    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606646 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.606418    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606646 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.606553    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.606772 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.606670    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.607092 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.607069    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.608387 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.608366    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.608604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.608588    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.627689 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.627658    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpmvw\" (UniqueName: \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-kube-api-access-bpmvw\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.628301 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.628271    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.708664 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.708622    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:40.974518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.974408    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 14:15:40.974518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:40.974487    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 14:15:41.078021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.077949    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"]
Nov 02 14:15:41.081633 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:15:41.081598    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76d8ed3_efd1_4e52_b0f8_ba73a77e0183.slice/crio-6a0c817361acaef02164c278a741c7febee76972c9440a269698c303137b9ae8 WatchSource:0}: Error finding container 6a0c817361acaef02164c278a741c7febee76972c9440a269698c303137b9ae8: Status 404 returned error can't find the container with id 6a0c817361acaef02164c278a741c7febee76972c9440a269698c303137b9ae8
Nov 02 14:15:41.084405 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.084369    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:15:41.084517 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.084467    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:15:41.084517 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.084503    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:15:41.140409 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.140362    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" event={"ID":"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183","Type":"ContainerStarted","Data":"6a0c817361acaef02164c278a741c7febee76972c9440a269698c303137b9ae8"}
Nov 02 14:15:41.142306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.142276    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc" event={"ID":"53e527d2-ba69-4c74-8d10-14d8fc00eeed","Type":"ContainerStarted","Data":"8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8"}
Nov 02 14:15:41.142464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.142314    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc" event={"ID":"53e527d2-ba69-4c74-8d10-14d8fc00eeed","Type":"ContainerStarted","Data":"3d7be958a03af014b8f7b8c3c56191247eeb686579aa9274bd87d96792dfb6f1"}
Nov 02 14:15:41.186373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.185197    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc" podStartSLOduration=2.185176674 podStartE2EDuration="2.185176674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:41.182440323 +0000 UTC m=+1873.141815931" watchObservedRunningTime="2026-11-02 14:15:41.185176674 +0000 UTC m=+1873.144552279"
Nov 02 14:15:41.720097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.720056    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:41.980461 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.980363    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Nov 02 14:15:41.982816 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:41.982779    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Nov 02 14:15:42.165958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:42.161177    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" event={"ID":"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183","Type":"ContainerStarted","Data":"256e06abd4347c3c130af60ec73300901e8c9c7f5903cdde0dfedfeba3448d02"}
Nov 02 14:15:42.225649 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:42.225587    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" podStartSLOduration=2.225570107 podStartE2EDuration="2.225570107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:42.21555293 +0000 UTC m=+1874.174928540" watchObservedRunningTime="2026-11-02 14:15:42.225570107 +0000 UTC m=+1874.184945715"
Nov 02 14:15:42.708966 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:42.708907    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:42.710894 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:42.710861    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 14:15:42.711063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:42.710915    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 14:15:43.172955 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:43.167200    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:43.172955 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:43.168755    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:43.181467 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:43.181436    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:15:43.709578 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:43.709544    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 14:15:43.709852 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:43.709829    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 14:15:43.904022 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:15:43.903982    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:15:44.709525 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:44.709490    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 14:15:44.710053 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:44.710023    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 14:15:45.709450 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:45.709407    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 14:15:45.709681 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:45.709479    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 14:15:46.709888 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:46.709848    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 14:15:46.710359 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:46.710036    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 14:15:47.714156 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:47.714117    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:47.714626 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:47.714546    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:47.715411 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:47.715392    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:15:49.719742 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:49.719704    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:49.735100 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:49.735063    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:15:50.248648 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:15:50.248613    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:16:01.874044 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:01.873953    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"]
Nov 02 14:16:01.880125 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:01.880101    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:01.899760 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:01.899722    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-5cbbfb-serving-cert\""
Nov 02 14:16:01.922810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:01.922767    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"]
Nov 02 14:16:01.951260 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:01.951203    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcxf\" (UniqueName: \"kubernetes.io/projected/12b411d4-f8a9-4ff9-b847-70a7dc138453-kube-api-access-xfcxf\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:01.951473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:01.951279    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-maas-api-tls\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:01.951473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:01.951310    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-metrics-certs\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:01.951473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:01.951437    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b411d4-f8a9-4ff9-b847-70a7dc138453-openshift-service-ca\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:02.052740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:02.052701    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcxf\" (UniqueName: \"kubernetes.io/projected/12b411d4-f8a9-4ff9-b847-70a7dc138453-kube-api-access-xfcxf\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:02.052740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:02.052751    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-maas-api-tls\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:02.053013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:02.052771    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-metrics-certs\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:02.053013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:02.052886    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b411d4-f8a9-4ff9-b847-70a7dc138453-openshift-service-ca\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:02.053013 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:16:02.052893    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-serving-cert: secret "maas-api-e2e-explicit-5cbbfb-serving-cert" not found
Nov 02 14:16:02.053013 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:16:02.052982    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-maas-api-tls podName:12b411d4-f8a9-4ff9-b847-70a7dc138453 nodeName:}" failed. No retries permitted until 2026-11-02 14:16:02.55295966 +0000 UTC m=+1894.512335263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-maas-api-tls") pod "maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf" (UID: "12b411d4-f8a9-4ff9-b847-70a7dc138453") : secret "maas-api-e2e-explicit-5cbbfb-serving-cert" not found
Nov 02 14:16:02.053629 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:02.053603    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b411d4-f8a9-4ff9-b847-70a7dc138453-openshift-service-ca\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:02.055464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:02.055439    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-metrics-certs\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:02.080984 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:02.080894    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcxf\" (UniqueName: \"kubernetes.io/projected/12b411d4-f8a9-4ff9-b847-70a7dc138453-kube-api-access-xfcxf\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:02.561377 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:02.561329    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-maas-api-tls\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:02.567725 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:02.567684    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-maas-api-tls\") pod \"maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:02.799581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:02.799549    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:03.188922 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:03.188195    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"]
Nov 02 14:16:03.214471 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:16:03.214371    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b411d4_f8a9_4ff9_b847_70a7dc138453.slice/crio-c4aad47a71fe2f34be4f4b7223dc9b3c6a1bbe06cce7ca10d66608c498b7f4db WatchSource:0}: Error finding container c4aad47a71fe2f34be4f4b7223dc9b3c6a1bbe06cce7ca10d66608c498b7f4db: Status 404 returned error can't find the container with id c4aad47a71fe2f34be4f4b7223dc9b3c6a1bbe06cce7ca10d66608c498b7f4db
Nov 02 14:16:03.333593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:03.333559    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf" event={"ID":"12b411d4-f8a9-4ff9-b847-70a7dc138453","Type":"ContainerStarted","Data":"c4aad47a71fe2f34be4f4b7223dc9b3c6a1bbe06cce7ca10d66608c498b7f4db"}
Nov 02 14:16:04.342751 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:04.342707    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf" event={"ID":"12b411d4-f8a9-4ff9-b847-70a7dc138453","Type":"ContainerStarted","Data":"00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20"}
Nov 02 14:16:04.343316 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:04.342785    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:04.381322 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:04.381127    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf" podStartSLOduration=3.041287341 podStartE2EDuration="3.381105986s" totalImagesPullingTime="339.818645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:04.38042803 +0000 UTC m=+1896.339803637" watchObservedRunningTime="2026-11-02 14:16:04.381105986 +0000 UTC m=+1896.340481592"
Nov 02 14:16:10.357604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:10.357567    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:16:13.897139 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:16:13.897083    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:16:31.860785 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.860750    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"]
Nov 02 14:16:31.866974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.866925    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:31.882525 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.882491    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"]
Nov 02 14:16:31.896208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.896164    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-21baa7\"/\"kube-root-ca.crt\""
Nov 02 14:16:31.896208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.896164    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-21baa7\"/\"openshift-service-ca.crt\""
Nov 02 14:16:31.897599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.897558    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-21baa7\"/\"default-dockercfg-kb7wh\""
Nov 02 14:16:31.897599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.897574    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-21baa7\"/\"auth-test-model-21baa7-kserve-self-signed-certs\""
Nov 02 14:16:31.976632 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.976597    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-home\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:31.976632 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.976647    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058eaa11-e58f-4867-80c0-fc1024b9805c-tls-certs\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:31.976958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.976682    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-dshm\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:31.976958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.976816    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-tmp-dir\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:31.976958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.976877    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-model-cache\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:31.977132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:31.977054    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bvbn\" (UniqueName: \"kubernetes.io/projected/058eaa11-e58f-4867-80c0-fc1024b9805c-kube-api-access-2bvbn\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.092991 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.088924    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bvbn\" (UniqueName: \"kubernetes.io/projected/058eaa11-e58f-4867-80c0-fc1024b9805c-kube-api-access-2bvbn\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.092991 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.089129    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-home\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.092991 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.089192    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058eaa11-e58f-4867-80c0-fc1024b9805c-tls-certs\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.092991 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.089229    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-dshm\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.092991 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.089308    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-tmp-dir\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.092991 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.089389    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-model-cache\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.092991 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.090319    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-home\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.096693 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.096660    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-model-cache\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.097238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.097213    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-tmp-dir\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.109855 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.109820    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-dshm\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.124101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.118355    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058eaa11-e58f-4867-80c0-fc1024b9805c-tls-certs\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.125094 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.125062    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bvbn\" (UniqueName: \"kubernetes.io/projected/058eaa11-e58f-4867-80c0-fc1024b9805c-kube-api-access-2bvbn\") pod \"auth-test-model-21baa7-kserve-b49b8df4f-rqszs\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.182187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.182155    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:32.520210 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.520177    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"]
Nov 02 14:16:32.522385 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:16:32.522359    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod058eaa11_e58f_4867_80c0_fc1024b9805c.slice/crio-c7424a4ea85844db20ac76f306e3daaae05df22375c2a37c4a7b38cc6f87a7e9 WatchSource:0}: Error finding container c7424a4ea85844db20ac76f306e3daaae05df22375c2a37c4a7b38cc6f87a7e9: Status 404 returned error can't find the container with id c7424a4ea85844db20ac76f306e3daaae05df22375c2a37c4a7b38cc6f87a7e9
Nov 02 14:16:32.568876 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:32.568831    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs" event={"ID":"058eaa11-e58f-4867-80c0-fc1024b9805c","Type":"ContainerStarted","Data":"c7424a4ea85844db20ac76f306e3daaae05df22375c2a37c4a7b38cc6f87a7e9"}
Nov 02 14:16:33.592827 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:33.589961    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs" event={"ID":"058eaa11-e58f-4867-80c0-fc1024b9805c","Type":"ContainerStarted","Data":"b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701"}
Nov 02 14:16:33.631819 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:33.631762    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs" podStartSLOduration=2.631743586 podStartE2EDuration="2.631743586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:33.627244044 +0000 UTC m=+1925.586619652" watchObservedRunningTime="2026-11-02 14:16:33.631743586 +0000 UTC m=+1925.591119193"
Nov 02 14:16:34.183324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:34.183288    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:42.183191 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:42.183145    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:42.196839 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:42.196808    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:42.663322 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:16:42.663284    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:16:43.900214 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:16:43.900172    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:17:13.896751 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:17:13.896707    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:17:26.546974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.546857    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"]
Nov 02 14:17:26.552026 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.551990    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.583334 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.583299    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"]
Nov 02 14:17:26.583583 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.583506    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-2daae4\"/\"kube-root-ca.crt\""
Nov 02 14:17:26.583689 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.583576    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-2daae4\"/\"openshift-service-ca.crt\""
Nov 02 14:17:26.583764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.583723    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-2daae4\"/\"default-dockercfg-ftq9x\""
Nov 02 14:17:26.583833 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.583795    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-2daae4\"/\"auth-test-model-2daae4-kserve-self-signed-certs\""
Nov 02 14:17:26.717274 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.717235    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt9sh\" (UniqueName: \"kubernetes.io/projected/fc00c131-1994-427e-91dd-b89632fa6720-kube-api-access-bt9sh\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.717476 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.717310    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc00c131-1994-427e-91dd-b89632fa6720-tls-certs\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.717476 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.717447    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-model-cache\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.717632 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.717615    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-tmp-dir\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.717674 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.717660    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-home\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.717824 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.717807    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-dshm\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.822177 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.822069    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt9sh\" (UniqueName: \"kubernetes.io/projected/fc00c131-1994-427e-91dd-b89632fa6720-kube-api-access-bt9sh\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.822955 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.822903    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc00c131-1994-427e-91dd-b89632fa6720-tls-certs\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.823222 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.823191    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-model-cache\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.823499 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.823483    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-tmp-dir\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.823652 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.823638    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-home\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.823829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.823795    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-model-cache\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.824016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.823862    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-dshm\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.824351 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.824321    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-tmp-dir\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.830962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.824612    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-home\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.873981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.873014    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt9sh\" (UniqueName: \"kubernetes.io/projected/fc00c131-1994-427e-91dd-b89632fa6720-kube-api-access-bt9sh\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.874502 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.874460    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-dshm\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.882019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.875816    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc00c131-1994-427e-91dd-b89632fa6720-tls-certs\") pod \"auth-test-model-2daae4-kserve-8545fc47c-h6pzl\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") " pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:26.897303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:26.897260    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:27.254171 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:27.254140    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"]
Nov 02 14:17:27.257182 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:17:27.257152    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc00c131_1994_427e_91dd_b89632fa6720.slice/crio-32e27ba758e343274ca1480de20639f98cfa485e6ecfd107b6e934d1f0d88993 WatchSource:0}: Error finding container 32e27ba758e343274ca1480de20639f98cfa485e6ecfd107b6e934d1f0d88993: Status 404 returned error can't find the container with id 32e27ba758e343274ca1480de20639f98cfa485e6ecfd107b6e934d1f0d88993
Nov 02 14:17:28.024376 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:28.024336    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl" event={"ID":"fc00c131-1994-427e-91dd-b89632fa6720","Type":"ContainerStarted","Data":"bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1"}
Nov 02 14:17:28.024884 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:28.024388    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl" event={"ID":"fc00c131-1994-427e-91dd-b89632fa6720","Type":"ContainerStarted","Data":"32e27ba758e343274ca1480de20639f98cfa485e6ecfd107b6e934d1f0d88993"}
Nov 02 14:17:28.063889 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:28.063826    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl" podStartSLOduration=2.06380397 podStartE2EDuration="2.06380397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:28.061770698 +0000 UTC m=+1980.021146304" watchObservedRunningTime="2026-11-02 14:17:28.06380397 +0000 UTC m=+1980.023179578"
Nov 02 14:17:28.898199 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:28.898157    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:36.897617 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:36.897573    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:36.911065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:36.911029    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:37.117844 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:17:37.117813    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:17:43.897633 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:17:43.897591    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:18:11.807788 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:11.807747    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"]
Nov 02 14:18:11.810192 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:11.808031    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf" podUID="12b411d4-f8a9-4ff9-b847-70a7dc138453" containerName="maas-api" containerID="cri-o://00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20" gracePeriod=30
Nov 02 14:18:12.369047 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.367873    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:18:12.401801 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.401761    6284 generic.go:356] "Generic (PLEG): container finished" podID="12b411d4-f8a9-4ff9-b847-70a7dc138453" containerID="00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20" exitCode=0
Nov 02 14:18:12.402054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.401826    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"
Nov 02 14:18:12.402054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.401846    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf" event={"ID":"12b411d4-f8a9-4ff9-b847-70a7dc138453","Type":"ContainerDied","Data":"00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20"}
Nov 02 14:18:12.402054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.401973    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf" event={"ID":"12b411d4-f8a9-4ff9-b847-70a7dc138453","Type":"ContainerDied","Data":"c4aad47a71fe2f34be4f4b7223dc9b3c6a1bbe06cce7ca10d66608c498b7f4db"}
Nov 02 14:18:12.402054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.402009    6284 scope.go:122] "RemoveContainer" containerID="00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20"
Nov 02 14:18:12.414924 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.414833    6284 scope.go:122] "RemoveContainer" containerID="00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20"
Nov 02 14:18:12.415614 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:18:12.415544    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20\": container with ID starting with 00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20 not found: ID does not exist" containerID="00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20"
Nov 02 14:18:12.415614 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.415586    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20"} err="failed to get container status \"00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20\": rpc error: code = NotFound desc = could not find container \"00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20\": container with ID starting with 00c3d6136c9dad12a9a2264c920d43927c3c0c123be83cfb6290616aedf18c20 not found: ID does not exist"
Nov 02 14:18:12.429256 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.429215    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b411d4-f8a9-4ff9-b847-70a7dc138453-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b411d4-f8a9-4ff9-b847-70a7dc138453-openshift-service-ca\") pod \"12b411d4-f8a9-4ff9-b847-70a7dc138453\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") "
Nov 02 14:18:12.429520 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.429450    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-metrics-certs\") pod \"12b411d4-f8a9-4ff9-b847-70a7dc138453\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") "
Nov 02 14:18:12.429520 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.429490    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b411d4-f8a9-4ff9-b847-70a7dc138453-kube-api-access-xfcxf\" (UniqueName: \"kubernetes.io/projected/12b411d4-f8a9-4ff9-b847-70a7dc138453-kube-api-access-xfcxf\") pod \"12b411d4-f8a9-4ff9-b847-70a7dc138453\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") "
Nov 02 14:18:12.429649 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.429622    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-maas-api-tls\") pod \"12b411d4-f8a9-4ff9-b847-70a7dc138453\" (UID: \"12b411d4-f8a9-4ff9-b847-70a7dc138453\") "
Nov 02 14:18:12.429916 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.429884    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b411d4-f8a9-4ff9-b847-70a7dc138453-openshift-service-ca" pod "12b411d4-f8a9-4ff9-b847-70a7dc138453" (UID: "12b411d4-f8a9-4ff9-b847-70a7dc138453"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:18:12.430138 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.430111    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b411d4-f8a9-4ff9-b847-70a7dc138453-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:18:12.432313 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.432281    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b411d4-f8a9-4ff9-b847-70a7dc138453-kube-api-access-xfcxf" pod "12b411d4-f8a9-4ff9-b847-70a7dc138453" (UID: "12b411d4-f8a9-4ff9-b847-70a7dc138453"). InnerVolumeSpecName "kube-api-access-xfcxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:18:12.432798 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.432771    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-metrics-certs" pod "12b411d4-f8a9-4ff9-b847-70a7dc138453" (UID: "12b411d4-f8a9-4ff9-b847-70a7dc138453"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:18:12.433925 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.433900    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-maas-api-tls" pod "12b411d4-f8a9-4ff9-b847-70a7dc138453" (UID: "12b411d4-f8a9-4ff9-b847-70a7dc138453"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:18:12.531264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.531224    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:18:12.531264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.531259    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfcxf\" (UniqueName: \"kubernetes.io/projected/12b411d4-f8a9-4ff9-b847-70a7dc138453-kube-api-access-xfcxf\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:18:12.531264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.531270    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12b411d4-f8a9-4ff9-b847-70a7dc138453-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:18:12.749228 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.749184    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"]
Nov 02 14:18:12.754906 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:12.754868    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5cbbfb-6f46b68f77-8jnpf"]
Nov 02 14:18:13.896366 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:18:13.896317    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:18:14.171551 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:14.171457    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b411d4-f8a9-4ff9-b847-70a7dc138453" path="/var/lib/kubelet/pods/12b411d4-f8a9-4ff9-b847-70a7dc138453/volumes"
Nov 02 14:18:26.150103 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.150059    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"]
Nov 02 14:18:26.150719 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.150702    6284 cpu_manager.go:414] "containerMap: removing container" podUID="12b411d4-f8a9-4ff9-b847-70a7dc138453" containerName="maas-api"
Nov 02 14:18:26.150769 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.150721    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b411d4-f8a9-4ff9-b847-70a7dc138453" containerName="maas-api"
Nov 02 14:18:26.150854 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.150842    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b411d4-f8a9-4ff9-b847-70a7dc138453" containerName="maas-api"
Nov 02 14:18:26.155912 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.155882    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.179891 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.179848    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-abc0f7\"/\"kube-root-ca.crt\""
Nov 02 14:18:26.180563 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.180543    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-abc0f7\"/\"noten-model-abc0f7-kserve-self-signed-certs\""
Nov 02 14:18:26.180954 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.179959    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"]
Nov 02 14:18:26.181148 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.181130    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-abc0f7\"/\"default-dockercfg-62n8c\""
Nov 02 14:18:26.181474 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.181453    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-abc0f7\"/\"openshift-service-ca.crt\""
Nov 02 14:18:26.255477 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.255251    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-model-cache\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.261178 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.261145    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-home\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.261259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.261228    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr972\" (UniqueName: \"kubernetes.io/projected/10f23f21-4838-46e0-8bd4-03973454b326-kube-api-access-dr972\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.261567 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.261545    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-tmp-dir\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.261700 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.261682    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f23f21-4838-46e0-8bd4-03973454b326-tls-certs\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.261796 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.261775    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-dshm\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.362698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.362661    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-tmp-dir\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.362698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.362711    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f23f21-4838-46e0-8bd4-03973454b326-tls-certs\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.363030 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.362755    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-dshm\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.363030 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.362823    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-model-cache\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.363030 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.362967    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-home\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.363030 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.363005    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr972\" (UniqueName: \"kubernetes.io/projected/10f23f21-4838-46e0-8bd4-03973454b326-kube-api-access-dr972\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.363318 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.363180    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-tmp-dir\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.363796 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.363430    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-model-cache\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.363925 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.363844    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-home\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.365897 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.365869    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f23f21-4838-46e0-8bd4-03973454b326-tls-certs\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.366051 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.365909    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-dshm\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.385735 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.385697    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr972\" (UniqueName: \"kubernetes.io/projected/10f23f21-4838-46e0-8bd4-03973454b326-kube-api-access-dr972\") pod \"noten-model-abc0f7-kserve-5898c55845-gm6vd\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") " pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.487957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.484134    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:26.850585 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:26.850550    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"]
Nov 02 14:18:26.852132 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:18:26.852084    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10f23f21_4838_46e0_8bd4_03973454b326.slice/crio-85d2c2f38455823b4b9f85bd4406ec8b35f56a18868833b045b91ca7be3ccf43 WatchSource:0}: Error finding container 85d2c2f38455823b4b9f85bd4406ec8b35f56a18868833b045b91ca7be3ccf43: Status 404 returned error can't find the container with id 85d2c2f38455823b4b9f85bd4406ec8b35f56a18868833b045b91ca7be3ccf43
Nov 02 14:18:27.577063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:27.577026    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd" event={"ID":"10f23f21-4838-46e0-8bd4-03973454b326","Type":"ContainerStarted","Data":"d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554"}
Nov 02 14:18:27.577063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:27.577072    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd" event={"ID":"10f23f21-4838-46e0-8bd4-03973454b326","Type":"ContainerStarted","Data":"85d2c2f38455823b4b9f85bd4406ec8b35f56a18868833b045b91ca7be3ccf43"}
Nov 02 14:18:27.615040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:27.614975    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd" podStartSLOduration=1.6149588129999999 podStartE2EDuration="1.614958813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:27.612710674 +0000 UTC m=+2039.572086281" watchObservedRunningTime="2026-11-02 14:18:27.614958813 +0000 UTC m=+2039.574334417"
Nov 02 14:18:28.485078 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:28.485044    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:36.485183 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:36.485145    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:36.498925 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:36.498892    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:36.679045 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:36.679007    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:18:43.897132 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:18:43.897091    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:18:51.228563 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:18:51.228344    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"]
Nov 02 14:19:06.254146 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.254104    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd" podUID="10f23f21-4838-46e0-8bd4-03973454b326" containerName="main" containerID="cri-o://d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554" gracePeriod=45
Nov 02 14:19:06.680210 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.680183    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:19:06.824611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.824550    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-tmp-dir\") pod \"10f23f21-4838-46e0-8bd4-03973454b326\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") "
Nov 02 14:19:06.825142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.825101    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10f23f21-4838-46e0-8bd4-03973454b326-tls-certs\" (UniqueName: \"kubernetes.io/secret/10f23f21-4838-46e0-8bd4-03973454b326-tls-certs\") pod \"10f23f21-4838-46e0-8bd4-03973454b326\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") "
Nov 02 14:19:06.825414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.825396    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f23f21-4838-46e0-8bd4-03973454b326-kube-api-access-dr972\" (UniqueName: \"kubernetes.io/projected/10f23f21-4838-46e0-8bd4-03973454b326-kube-api-access-dr972\") pod \"10f23f21-4838-46e0-8bd4-03973454b326\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") "
Nov 02 14:19:06.825586 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.825571    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-home\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-home\") pod \"10f23f21-4838-46e0-8bd4-03973454b326\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") "
Nov 02 14:19:06.825868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.825835    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-model-cache\") pod \"10f23f21-4838-46e0-8bd4-03973454b326\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") "
Nov 02 14:19:06.826072 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.826055    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-dshm\") pod \"10f23f21-4838-46e0-8bd4-03973454b326\" (UID: \"10f23f21-4838-46e0-8bd4-03973454b326\") "
Nov 02 14:19:06.826270 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.826239    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-home" pod "10f23f21-4838-46e0-8bd4-03973454b326" (UID: "10f23f21-4838-46e0-8bd4-03973454b326"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:06.826979 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.826953    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:06.828920 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.828881    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-tmp-dir" pod "10f23f21-4838-46e0-8bd4-03973454b326" (UID: "10f23f21-4838-46e0-8bd4-03973454b326"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:06.830251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.830208    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-dshm" pod "10f23f21-4838-46e0-8bd4-03973454b326" (UID: "10f23f21-4838-46e0-8bd4-03973454b326"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:06.834542 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.834501    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f23f21-4838-46e0-8bd4-03973454b326-tls-certs" pod "10f23f21-4838-46e0-8bd4-03973454b326" (UID: "10f23f21-4838-46e0-8bd4-03973454b326"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:19:06.844363 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.835711    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-model-cache" pod "10f23f21-4838-46e0-8bd4-03973454b326" (UID: "10f23f21-4838-46e0-8bd4-03973454b326"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:06.864258 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.863211    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f23f21-4838-46e0-8bd4-03973454b326-kube-api-access-dr972" pod "10f23f21-4838-46e0-8bd4-03973454b326" (UID: "10f23f21-4838-46e0-8bd4-03973454b326"). InnerVolumeSpecName "kube-api-access-dr972". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:06.932039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.928284    6284 generic.go:356] "Generic (PLEG): container finished" podID="10f23f21-4838-46e0-8bd4-03973454b326" containerID="d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554" exitCode=0
Nov 02 14:19:06.932039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.928492    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd" event={"ID":"10f23f21-4838-46e0-8bd4-03973454b326","Type":"ContainerDied","Data":"d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554"}
Nov 02 14:19:06.932039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.928533    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd" event={"ID":"10f23f21-4838-46e0-8bd4-03973454b326","Type":"ContainerDied","Data":"85d2c2f38455823b4b9f85bd4406ec8b35f56a18868833b045b91ca7be3ccf43"}
Nov 02 14:19:06.932039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.928558    6284 scope.go:122] "RemoveContainer" containerID="d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554"
Nov 02 14:19:06.932039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.928725    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"
Nov 02 14:19:06.943034 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.942902    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:06.943294 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.943220    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:06.943294 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.943243    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10f23f21-4838-46e0-8bd4-03973454b326-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:06.943294 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.943256    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr972\" (UniqueName: \"kubernetes.io/projected/10f23f21-4838-46e0-8bd4-03973454b326-kube-api-access-dr972\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:06.943294 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.943271    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10f23f21-4838-46e0-8bd4-03973454b326-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:06.961527 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.961153    6284 scope.go:122] "RemoveContainer" containerID="d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554"
Nov 02 14:19:06.961925 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:19:06.961824    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554\": container with ID starting with d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554 not found: ID does not exist" containerID="d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554"
Nov 02 14:19:06.961925 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.961871    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554"} err="failed to get container status \"d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554\": rpc error: code = NotFound desc = could not find container \"d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554\": container with ID starting with d8e6f7cb89af7b1bc9e0849f7687104504f42f450a626f9de750292e2cec1554 not found: ID does not exist"
Nov 02 14:19:06.987791 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.987745    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"]
Nov 02 14:19:06.996249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:06.996210    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd"]
Nov 02 14:19:07.669115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:07.669040    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-abc0f7/noten-model-abc0f7-kserve-5898c55845-gm6vd" podUID="10f23f21-4838-46e0-8bd4-03973454b326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.116:8000/ready\": context deadline exceeded"
Nov 02 14:19:08.172463 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:08.172426    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f23f21-4838-46e0-8bd4-03973454b326" path="/var/lib/kubelet/pods/10f23f21-4838-46e0-8bd4-03973454b326/volumes"
Nov 02 14:19:13.895557 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:19:13.895517    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:19:28.770311 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:28.770278    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:19:28.775089 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:28.775059    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:19:33.262824 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:33.262781    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"]
Nov 02 14:19:43.897008 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:19:43.896963    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:19:47.738807 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:47.738766    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"]
Nov 02 14:19:47.739254 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:47.739002    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy" containerID="cri-o://4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6" gracePeriod=30
Nov 02 14:19:49.158080 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.158013    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc" podUID="53e527d2-ba69-4c74-8d10-14d8fc00eeed" containerName="main" containerID="cri-o://8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8" gracePeriod=45
Nov 02 14:19:49.236838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.236793    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc" podUID="53e527d2-ba69-4c74-8d10-14d8fc00eeed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.111:8000/ready\": dial tcp 10.134.0.111:8000: connect: connection refused"
Nov 02 14:19:49.455406 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.455372    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:19:49.536298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.534801    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tls-certs\" (UniqueName: \"kubernetes.io/secret/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tls-certs\") pod \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") "
Nov 02 14:19:49.536298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.534982    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tmp-dir\") pod \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") "
Nov 02 14:19:49.536298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.535127    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-home\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-home\") pod \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") "
Nov 02 14:19:49.536298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.535160    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-model-cache\") pod \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") "
Nov 02 14:19:49.536298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.535243    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e527d2-ba69-4c74-8d10-14d8fc00eeed-kube-api-access-lhmj8\" (UniqueName: \"kubernetes.io/projected/53e527d2-ba69-4c74-8d10-14d8fc00eeed-kube-api-access-lhmj8\") pod \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") "
Nov 02 14:19:49.536298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.535342    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-dshm\") pod \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\" (UID: \"53e527d2-ba69-4c74-8d10-14d8fc00eeed\") "
Nov 02 14:19:49.536849 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.536472    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-home" pod "53e527d2-ba69-4c74-8d10-14d8fc00eeed" (UID: "53e527d2-ba69-4c74-8d10-14d8fc00eeed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:49.537507 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.537465    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-model-cache" pod "53e527d2-ba69-4c74-8d10-14d8fc00eeed" (UID: "53e527d2-ba69-4c74-8d10-14d8fc00eeed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:49.537870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.537850    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tmp-dir" pod "53e527d2-ba69-4c74-8d10-14d8fc00eeed" (UID: "53e527d2-ba69-4c74-8d10-14d8fc00eeed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:49.546090 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.545484    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-dshm" pod "53e527d2-ba69-4c74-8d10-14d8fc00eeed" (UID: "53e527d2-ba69-4c74-8d10-14d8fc00eeed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:49.546090 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.545499    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tls-certs" pod "53e527d2-ba69-4c74-8d10-14d8fc00eeed" (UID: "53e527d2-ba69-4c74-8d10-14d8fc00eeed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:19:49.547066 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.547036    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e527d2-ba69-4c74-8d10-14d8fc00eeed-kube-api-access-lhmj8" pod "53e527d2-ba69-4c74-8d10-14d8fc00eeed" (UID: "53e527d2-ba69-4c74-8d10-14d8fc00eeed"). InnerVolumeSpecName "kube-api-access-lhmj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:49.637609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.637252    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:49.637609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.637294    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:49.637609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.637312    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhmj8\" (UniqueName: \"kubernetes.io/projected/53e527d2-ba69-4c74-8d10-14d8fc00eeed-kube-api-access-lhmj8\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:49.637609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.637327    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:49.637609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.637341    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:49.637609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:49.637383    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53e527d2-ba69-4c74-8d10-14d8fc00eeed-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:50.258746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:50.258707    6284 generic.go:356] "Generic (PLEG): container finished" podID="53e527d2-ba69-4c74-8d10-14d8fc00eeed" containerID="8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8" exitCode=0
Nov 02 14:19:50.259272 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:50.258753    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc" event={"ID":"53e527d2-ba69-4c74-8d10-14d8fc00eeed","Type":"ContainerDied","Data":"8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8"}
Nov 02 14:19:50.259272 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:50.258785    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc" event={"ID":"53e527d2-ba69-4c74-8d10-14d8fc00eeed","Type":"ContainerDied","Data":"3d7be958a03af014b8f7b8c3c56191247eeb686579aa9274bd87d96792dfb6f1"}
Nov 02 14:19:50.259272 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:50.258804    6284 scope.go:122] "RemoveContainer" containerID="8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8"
Nov 02 14:19:50.259272 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:50.258757    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"
Nov 02 14:19:50.269861 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:50.269836    6284 scope.go:122] "RemoveContainer" containerID="8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8"
Nov 02 14:19:50.270267 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:19:50.270243    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8\": container with ID starting with 8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8 not found: ID does not exist" containerID="8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8"
Nov 02 14:19:50.270349 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:50.270284    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8"} err="failed to get container status \"8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8\": rpc error: code = NotFound desc = could not find container \"8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8\": container with ID starting with 8f5ae04dcd79426bafc4de03000b0ce4f671fa9b06d8930d2f80049073cef9c8 not found: ID does not exist"
Nov 02 14:19:50.306283 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:50.305256    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"]
Nov 02 14:19:50.314193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:50.314158    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/auth-test-model-2cddf2-kserve-7c5b47cd69-ztlbc"]
Nov 02 14:19:52.170531 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:52.170494    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e527d2-ba69-4c74-8d10-14d8fc00eeed" path="/var/lib/kubelet/pods/53e527d2-ba69-4c74-8d10-14d8fc00eeed/volumes"
Nov 02 14:19:53.025952 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.025899    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:19:53.091609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.091578    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-envoy\") pod \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") "
Nov 02 14:19:53.091806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.091705    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-token\" (UniqueName: \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-token\") pod \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") "
Nov 02 14:19:53.091806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.091737    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-credential-socket\") pod \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") "
Nov 02 14:19:53.091984 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.091811    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-socket\") pod \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") "
Nov 02 14:19:53.091984 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.091879    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-data\") pod \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") "
Nov 02 14:19:53.091984 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.091906    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-podinfo\") pod \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") "
Nov 02 14:19:53.091984 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.091955    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-kube-api-access-v478l\" (UniqueName: \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-kube-api-access-v478l\") pod \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") "
Nov 02 14:19:53.092200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.092012    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-certs\") pod \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") "
Nov 02 14:19:53.092200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.092083    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istiod-ca-cert\") pod \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\" (UID: \"4e13ec12-9d79-4a64-ba6d-914250c86d2d\") "
Nov 02 14:19:53.092311 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.092278    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-data" pod "4e13ec12-9d79-4a64-ba6d-914250c86d2d" (UID: "4e13ec12-9d79-4a64-ba6d-914250c86d2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:53.092772 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.092750    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:53.093618 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.093287    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istiod-ca-cert" pod "4e13ec12-9d79-4a64-ba6d-914250c86d2d" (UID: "4e13ec12-9d79-4a64-ba6d-914250c86d2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:19:53.093618 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.093328    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-credential-socket" pod "4e13ec12-9d79-4a64-ba6d-914250c86d2d" (UID: "4e13ec12-9d79-4a64-ba6d-914250c86d2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:53.093828 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.093536    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-certs" pod "4e13ec12-9d79-4a64-ba6d-914250c86d2d" (UID: "4e13ec12-9d79-4a64-ba6d-914250c86d2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:53.094756 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.094727    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-socket" pod "4e13ec12-9d79-4a64-ba6d-914250c86d2d" (UID: "4e13ec12-9d79-4a64-ba6d-914250c86d2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:53.095622 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.095575    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-envoy" pod "4e13ec12-9d79-4a64-ba6d-914250c86d2d" (UID: "4e13ec12-9d79-4a64-ba6d-914250c86d2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:53.095728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.095627    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-token" pod "4e13ec12-9d79-4a64-ba6d-914250c86d2d" (UID: "4e13ec12-9d79-4a64-ba6d-914250c86d2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:53.095728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.095662    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-kube-api-access-v478l" pod "4e13ec12-9d79-4a64-ba6d-914250c86d2d" (UID: "4e13ec12-9d79-4a64-ba6d-914250c86d2d"). InnerVolumeSpecName "kube-api-access-v478l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:53.096725 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.096697    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-podinfo" pod "4e13ec12-9d79-4a64-ba6d-914250c86d2d" (UID: "4e13ec12-9d79-4a64-ba6d-914250c86d2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:19:53.197981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.196233    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:53.197981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.196274    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:53.197981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.196290    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:53.197981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.196304    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:53.197981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.196317    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v478l\" (UniqueName: \"kubernetes.io/projected/4e13ec12-9d79-4a64-ba6d-914250c86d2d-kube-api-access-v478l\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:53.197981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.196330    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:53.197981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.196343    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:53.197981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.196358    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e13ec12-9d79-4a64-ba6d-914250c86d2d-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:19:53.281317 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.281223    6284 generic.go:356] "Generic (PLEG): container finished" podID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerID="4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6" exitCode=0
Nov 02 14:19:53.281317 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.281279    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"
Nov 02 14:19:53.281536 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.281319    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" event={"ID":"4e13ec12-9d79-4a64-ba6d-914250c86d2d","Type":"ContainerDied","Data":"4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6"}
Nov 02 14:19:53.281536 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.281379    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs" event={"ID":"4e13ec12-9d79-4a64-ba6d-914250c86d2d","Type":"ContainerDied","Data":"20523697a5e40ee86ff349a2d872ede4174d911a94b17040eb5b3a30d669e94a"}
Nov 02 14:19:53.281536 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.281406    6284 scope.go:122] "RemoveContainer" containerID="4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6"
Nov 02 14:19:53.293716 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.293689    6284 scope.go:122] "RemoveContainer" containerID="4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6"
Nov 02 14:19:53.294126 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:19:53.294101    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6\": container with ID starting with 4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6 not found: ID does not exist" containerID="4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6"
Nov 02 14:19:53.294254 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.294135    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6"} err="failed to get container status \"4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6\": rpc error: code = NotFound desc = could not find container \"4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6\": container with ID starting with 4154dc4641d43081f48adb74be90a85666bb5c04ab027538b670802cd86c55f6 not found: ID does not exist"
Nov 02 14:19:53.334885 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.334839    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"]
Nov 02 14:19:53.340568 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:53.340534    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-dl9zs"]
Nov 02 14:19:54.174050 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:19:54.174010    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" path="/var/lib/kubelet/pods/4e13ec12-9d79-4a64-ba6d-914250c86d2d/volumes"
Nov 02 14:20:13.900315 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:20:13.900266    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:20:16.857993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:16.857948    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"]
Nov 02 14:20:16.858510 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:16.858190    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy" containerID="cri-o://f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48" gracePeriod=30
Nov 02 14:20:22.130374 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.130299    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 14:20:22.246071 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246034    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-kube-api-access-4hfkv\" (UniqueName: \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-kube-api-access-4hfkv\") pod \"907e32be-943b-47c5-8091-2bc94b31f433\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") "
Nov 02 14:20:22.246248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246105    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-certs\") pod \"907e32be-943b-47c5-8091-2bc94b31f433\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") "
Nov 02 14:20:22.246248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246134    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907e32be-943b-47c5-8091-2bc94b31f433-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907e32be-943b-47c5-8091-2bc94b31f433-istiod-ca-cert\") pod \"907e32be-943b-47c5-8091-2bc94b31f433\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") "
Nov 02 14:20:22.246248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246182    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-socket\") pod \"907e32be-943b-47c5-8091-2bc94b31f433\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") "
Nov 02 14:20:22.246248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246225    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-envoy\") pod \"907e32be-943b-47c5-8091-2bc94b31f433\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") "
Nov 02 14:20:22.246437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246255    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-credential-socket\") pod \"907e32be-943b-47c5-8091-2bc94b31f433\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") "
Nov 02 14:20:22.246437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246310    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-istio-token\" (UniqueName: \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-istio-token\") pod \"907e32be-943b-47c5-8091-2bc94b31f433\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") "
Nov 02 14:20:22.246437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246372    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/907e32be-943b-47c5-8091-2bc94b31f433-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907e32be-943b-47c5-8091-2bc94b31f433-istio-podinfo\") pod \"907e32be-943b-47c5-8091-2bc94b31f433\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") "
Nov 02 14:20:22.246606 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246465    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-data\") pod \"907e32be-943b-47c5-8091-2bc94b31f433\" (UID: \"907e32be-943b-47c5-8091-2bc94b31f433\") "
Nov 02 14:20:22.246606 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246472    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-certs" pod "907e32be-943b-47c5-8091-2bc94b31f433" (UID: "907e32be-943b-47c5-8091-2bc94b31f433"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:22.246606 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246507    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-socket" pod "907e32be-943b-47c5-8091-2bc94b31f433" (UID: "907e32be-943b-47c5-8091-2bc94b31f433"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:22.246606 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246571    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907e32be-943b-47c5-8091-2bc94b31f433-istiod-ca-cert" pod "907e32be-943b-47c5-8091-2bc94b31f433" (UID: "907e32be-943b-47c5-8091-2bc94b31f433"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:22.246866 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246799    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-data" pod "907e32be-943b-47c5-8091-2bc94b31f433" (UID: "907e32be-943b-47c5-8091-2bc94b31f433"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:22.246993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.246925    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-credential-socket" pod "907e32be-943b-47c5-8091-2bc94b31f433" (UID: "907e32be-943b-47c5-8091-2bc94b31f433"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:22.247128 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.247048    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:22.247128 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.247063    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:22.247128 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.247076    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/907e32be-943b-47c5-8091-2bc94b31f433-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:22.247128 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.247090    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:22.248518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.248484    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-envoy" pod "907e32be-943b-47c5-8091-2bc94b31f433" (UID: "907e32be-943b-47c5-8091-2bc94b31f433"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:22.248734 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.248705    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-istio-token" pod "907e32be-943b-47c5-8091-2bc94b31f433" (UID: "907e32be-943b-47c5-8091-2bc94b31f433"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:22.248833 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.248805    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/907e32be-943b-47c5-8091-2bc94b31f433-istio-podinfo" pod "907e32be-943b-47c5-8091-2bc94b31f433" (UID: "907e32be-943b-47c5-8091-2bc94b31f433"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:20:22.248921 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.248829    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-kube-api-access-4hfkv" pod "907e32be-943b-47c5-8091-2bc94b31f433" (UID: "907e32be-943b-47c5-8091-2bc94b31f433"). InnerVolumeSpecName "kube-api-access-4hfkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:22.348442 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.348404    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:22.348442 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.348436    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/907e32be-943b-47c5-8091-2bc94b31f433-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:22.348442 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.348446    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:22.348442 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.348454    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/907e32be-943b-47c5-8091-2bc94b31f433-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:22.348733 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.348465    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hfkv\" (UniqueName: \"kubernetes.io/projected/907e32be-943b-47c5-8091-2bc94b31f433-kube-api-access-4hfkv\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:22.503681 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.503646    6284 generic.go:356] "Generic (PLEG): container finished" podID="907e32be-943b-47c5-8091-2bc94b31f433" containerID="f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48" exitCode=0
Nov 02 14:20:22.503948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.503692    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"
Nov 02 14:20:22.503948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.503731    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" event={"ID":"907e32be-943b-47c5-8091-2bc94b31f433","Type":"ContainerDied","Data":"f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48"}
Nov 02 14:20:22.503948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.503790    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz" event={"ID":"907e32be-943b-47c5-8091-2bc94b31f433","Type":"ContainerDied","Data":"1617d3e5ecbb7394234af84ddba08036a25345d60262606aaea0d07eb1a68368"}
Nov 02 14:20:22.503948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.503816    6284 scope.go:122] "RemoveContainer" containerID="f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48"
Nov 02 14:20:22.515859 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.515836    6284 scope.go:122] "RemoveContainer" containerID="f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48"
Nov 02 14:20:22.516228 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:20:22.516206    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48\": container with ID starting with f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48 not found: ID does not exist" containerID="f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48"
Nov 02 14:20:22.516331 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.516237    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48"} err="failed to get container status \"f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48\": rpc error: code = NotFound desc = could not find container \"f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48\": container with ID starting with f78569a8afcd6ace56f5664c48b746c33d9673f5983e66271fb171f5ddc1bc48 not found: ID does not exist"
Nov 02 14:20:22.550897 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.550858    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"]
Nov 02 14:20:22.560868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:22.560832    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-bptpz"]
Nov 02 14:20:24.169834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:24.169795    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907e32be-943b-47c5-8091-2bc94b31f433" path="/var/lib/kubelet/pods/907e32be-943b-47c5-8091-2bc94b31f433/volumes"
Nov 02 14:20:43.895110 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:20:43.895062    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:20:53.270358 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:53.270316    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"]
Nov 02 14:20:53.271038 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:53.270555    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy" containerID="cri-o://f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d" gracePeriod=30
Nov 02 14:20:58.583179 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.583147    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:20:58.650868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.650790    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-kube-api-access-9fsll\" (UniqueName: \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-kube-api-access-9fsll\") pod \"982cc2a7-4243-406e-be09-462c0dd2d06a\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") "
Nov 02 14:20:58.651087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.650905    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-credential-socket\") pod \"982cc2a7-4243-406e-be09-462c0dd2d06a\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") "
Nov 02 14:20:58.651087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.650985    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-podinfo\") pod \"982cc2a7-4243-406e-be09-462c0dd2d06a\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") "
Nov 02 14:20:58.651087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.651035    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982cc2a7-4243-406e-be09-462c0dd2d06a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982cc2a7-4243-406e-be09-462c0dd2d06a-istiod-ca-cert\") pod \"982cc2a7-4243-406e-be09-462c0dd2d06a\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") "
Nov 02 14:20:58.651267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.651109    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-data\") pod \"982cc2a7-4243-406e-be09-462c0dd2d06a\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") "
Nov 02 14:20:58.651267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.651205    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-token\" (UniqueName: \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-token\") pod \"982cc2a7-4243-406e-be09-462c0dd2d06a\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") "
Nov 02 14:20:58.651267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.651248    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-envoy\") pod \"982cc2a7-4243-406e-be09-462c0dd2d06a\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") "
Nov 02 14:20:58.651414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.651276    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-credential-socket" pod "982cc2a7-4243-406e-be09-462c0dd2d06a" (UID: "982cc2a7-4243-406e-be09-462c0dd2d06a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:58.651414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.651309    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-socket\") pod \"982cc2a7-4243-406e-be09-462c0dd2d06a\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") "
Nov 02 14:20:58.651414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.651365    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-certs\") pod \"982cc2a7-4243-406e-be09-462c0dd2d06a\" (UID: \"982cc2a7-4243-406e-be09-462c0dd2d06a\") "
Nov 02 14:20:58.651569 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.651473    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-data" pod "982cc2a7-4243-406e-be09-462c0dd2d06a" (UID: "982cc2a7-4243-406e-be09-462c0dd2d06a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:58.652120 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.651790    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982cc2a7-4243-406e-be09-462c0dd2d06a-istiod-ca-cert" pod "982cc2a7-4243-406e-be09-462c0dd2d06a" (UID: "982cc2a7-4243-406e-be09-462c0dd2d06a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:58.652120 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.652008    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/982cc2a7-4243-406e-be09-462c0dd2d06a-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:58.652120 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.652028    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:58.652120 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.652041    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:58.652120 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.652089    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-socket" pod "982cc2a7-4243-406e-be09-462c0dd2d06a" (UID: "982cc2a7-4243-406e-be09-462c0dd2d06a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:58.653151 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.653108    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-certs" pod "982cc2a7-4243-406e-be09-462c0dd2d06a" (UID: "982cc2a7-4243-406e-be09-462c0dd2d06a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:58.653917 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.653879    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-podinfo" pod "982cc2a7-4243-406e-be09-462c0dd2d06a" (UID: "982cc2a7-4243-406e-be09-462c0dd2d06a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:20:58.657453 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.657415    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-envoy" pod "982cc2a7-4243-406e-be09-462c0dd2d06a" (UID: "982cc2a7-4243-406e-be09-462c0dd2d06a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:58.657997 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.657456    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-kube-api-access-9fsll" pod "982cc2a7-4243-406e-be09-462c0dd2d06a" (UID: "982cc2a7-4243-406e-be09-462c0dd2d06a"). InnerVolumeSpecName "kube-api-access-9fsll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:58.660160 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.660125    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-token" pod "982cc2a7-4243-406e-be09-462c0dd2d06a" (UID: "982cc2a7-4243-406e-be09-462c0dd2d06a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:58.753195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.753159    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:58.753195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.753190    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:58.753195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.753200    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:58.753458 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.753210    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/982cc2a7-4243-406e-be09-462c0dd2d06a-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:58.753458 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.753220    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fsll\" (UniqueName: \"kubernetes.io/projected/982cc2a7-4243-406e-be09-462c0dd2d06a-kube-api-access-9fsll\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:58.753458 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.753228    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/982cc2a7-4243-406e-be09-462c0dd2d06a-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:20:58.889105 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.888993    6284 generic.go:356] "Generic (PLEG): container finished" podID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerID="f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d" exitCode=0
Nov 02 14:20:58.889105 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.889045    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"
Nov 02 14:20:58.889105 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.889055    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" event={"ID":"982cc2a7-4243-406e-be09-462c0dd2d06a","Type":"ContainerDied","Data":"f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d"}
Nov 02 14:20:58.889400 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.889147    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j" event={"ID":"982cc2a7-4243-406e-be09-462c0dd2d06a","Type":"ContainerDied","Data":"5233af024d98ad7b0286b8e84da34d965fdb2a48cc7d5c612a20d1fd080ddf20"}
Nov 02 14:20:58.889400 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.889184    6284 scope.go:122] "RemoveContainer" containerID="f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d"
Nov 02 14:20:58.900852 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.900827    6284 scope.go:122] "RemoveContainer" containerID="f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d"
Nov 02 14:20:58.901372 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:20:58.901344    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d\": container with ID starting with f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d not found: ID does not exist" containerID="f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d"
Nov 02 14:20:58.901520 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.901381    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d"} err="failed to get container status \"f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d\": rpc error: code = NotFound desc = could not find container \"f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d\": container with ID starting with f2c865e3c783d7519af2307d7a5739c5ef303a0523c33d50a07b44d0a909dc0d not found: ID does not exist"
Nov 02 14:20:58.935558 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.935519    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"]
Nov 02 14:20:58.945010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:58.944971    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-rvs8j"]
Nov 02 14:20:59.751709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:20:59.751666    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"]
Nov 02 14:21:00.170089 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:00.170009    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" path="/var/lib/kubelet/pods/982cc2a7-4243-406e-be09-462c0dd2d06a/volumes"
Nov 02 14:21:02.157613 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:02.157573    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"]
Nov 02 14:21:02.158076 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:02.157844    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy" containerID="cri-o://a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731" gracePeriod=30
Nov 02 14:21:07.442539 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.442459    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 14:21:07.577518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.577478    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-envoy\") pod \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") "
Nov 02 14:21:07.577701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.577572    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-credential-socket\") pod \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") "
Nov 02 14:21:07.577701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.577629    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-podinfo\") pod \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") "
Nov 02 14:21:07.577842 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.577723    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-certs\") pod \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") "
Nov 02 14:21:07.577842 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.577750    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-data\") pod \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") "
Nov 02 14:21:07.577842 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.577798    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-socket\") pod \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") "
Nov 02 14:21:07.578060 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.577861    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-credential-socket" pod "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" (UID: "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:07.578060 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.577889    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-kube-api-access-66nd8\" (UniqueName: \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-kube-api-access-66nd8\") pod \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") "
Nov 02 14:21:07.578060 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.577974    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-token\" (UniqueName: \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-token\") pod \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") "
Nov 02 14:21:07.578060 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.578034    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istiod-ca-cert\") pod \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\" (UID: \"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd\") "
Nov 02 14:21:07.578259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.578068    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-data" pod "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" (UID: "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:07.578259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.578082    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-certs" pod "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" (UID: "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:07.578259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.578150    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-socket" pod "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" (UID: "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:07.578506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.578481    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istiod-ca-cert" pod "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" (UID: "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:21:07.578574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.578560    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:07.578636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.578597    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:07.578636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.578614    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:07.578636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.578627    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:07.580652 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.580613    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-envoy" pod "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" (UID: "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:07.580806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.580663    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-kube-api-access-66nd8" pod "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" (UID: "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd"). InnerVolumeSpecName "kube-api-access-66nd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:21:07.580806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.580762    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-token" pod "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" (UID: "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:21:07.580895 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.580837    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-podinfo" pod "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" (UID: "dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:21:07.679842 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.679790    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66nd8\" (UniqueName: \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-kube-api-access-66nd8\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:07.679842 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.679835    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:07.679842 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.679844    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:07.679842 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.679853    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:07.680182 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:07.679864    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:08.011223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:08.011191    6284 generic.go:356] "Generic (PLEG): container finished" podID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerID="a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731" exitCode=0
Nov 02 14:21:08.011416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:08.011242    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"
Nov 02 14:21:08.011416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:08.011283    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" event={"ID":"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd","Type":"ContainerDied","Data":"a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731"}
Nov 02 14:21:08.011416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:08.011352    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4" event={"ID":"dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd","Type":"ContainerDied","Data":"747888e8ebc8e1f386eb85919adb034a627f0420f13e4e9c5041d5127fb0c173"}
Nov 02 14:21:08.011416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:08.011377    6284 scope.go:122] "RemoveContainer" containerID="a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731"
Nov 02 14:21:08.036782 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:08.036751    6284 scope.go:122] "RemoveContainer" containerID="a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731"
Nov 02 14:21:08.037224 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:21:08.037195    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731\": container with ID starting with a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731 not found: ID does not exist" containerID="a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731"
Nov 02 14:21:08.037425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:08.037387    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731"} err="failed to get container status \"a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731\": rpc error: code = NotFound desc = could not find container \"a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731\": container with ID starting with a5f9f64fc771663d6c53ca290b776db4133cf2336d77ee71783e2ebee3448731 not found: ID does not exist"
Nov 02 14:21:08.089738 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:08.089687    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"]
Nov 02 14:21:08.092702 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:08.092669    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-5j2t4"]
Nov 02 14:21:08.169983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:08.169918    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" path="/var/lib/kubelet/pods/dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd/volumes"
Nov 02 14:21:13.897497 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:21:13.897446    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:21:14.218296 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:14.218196    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"]
Nov 02 14:21:14.218475 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:14.218419    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" containerName="istio-proxy" containerID="cri-o://116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3" gracePeriod=30
Nov 02 14:21:14.788625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:14.788579    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl" podUID="fc00c131-1994-427e-91dd-b89632fa6720" containerName="main" containerID="cri-o://bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1" gracePeriod=45
Nov 02 14:21:15.076618 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.076583    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:21:15.081025 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.080989    6284 generic.go:356] "Generic (PLEG): container finished" podID="fc00c131-1994-427e-91dd-b89632fa6720" containerID="bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1" exitCode=0
Nov 02 14:21:15.081153 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.081058    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"
Nov 02 14:21:15.081153 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.081078    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl" event={"ID":"fc00c131-1994-427e-91dd-b89632fa6720","Type":"ContainerDied","Data":"bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1"}
Nov 02 14:21:15.081153 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.081131    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl" event={"ID":"fc00c131-1994-427e-91dd-b89632fa6720","Type":"ContainerDied","Data":"32e27ba758e343274ca1480de20639f98cfa485e6ecfd107b6e934d1f0d88993"}
Nov 02 14:21:15.081285 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.081155    6284 scope.go:122] "RemoveContainer" containerID="bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1"
Nov 02 14:21:15.092051 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.092026    6284 scope.go:122] "RemoveContainer" containerID="bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1"
Nov 02 14:21:15.092398 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:21:15.092371    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1\": container with ID starting with bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1 not found: ID does not exist" containerID="bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1"
Nov 02 14:21:15.092486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.092414    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1"} err="failed to get container status \"bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1\": rpc error: code = NotFound desc = could not find container \"bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1\": container with ID starting with bbf283e409997d53dd5a2e28c8d8d5989e1189a211af233972cb041a60e676f1 not found: ID does not exist"
Nov 02 14:21:15.120783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.120746    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc00c131-1994-427e-91dd-b89632fa6720-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc00c131-1994-427e-91dd-b89632fa6720-tls-certs\") pod \"fc00c131-1994-427e-91dd-b89632fa6720\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") "
Nov 02 14:21:15.121034 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.120842    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-dshm\") pod \"fc00c131-1994-427e-91dd-b89632fa6720\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") "
Nov 02 14:21:15.122013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.121338    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-tmp-dir\") pod \"fc00c131-1994-427e-91dd-b89632fa6720\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") "
Nov 02 14:21:15.122013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.121458    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc00c131-1994-427e-91dd-b89632fa6720-kube-api-access-bt9sh\" (UniqueName: \"kubernetes.io/projected/fc00c131-1994-427e-91dd-b89632fa6720-kube-api-access-bt9sh\") pod \"fc00c131-1994-427e-91dd-b89632fa6720\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") "
Nov 02 14:21:15.122013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.121489    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-home\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-home\") pod \"fc00c131-1994-427e-91dd-b89632fa6720\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") "
Nov 02 14:21:15.122013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.121565    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-model-cache\") pod \"fc00c131-1994-427e-91dd-b89632fa6720\" (UID: \"fc00c131-1994-427e-91dd-b89632fa6720\") "
Nov 02 14:21:15.123250 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.123218    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-home" pod "fc00c131-1994-427e-91dd-b89632fa6720" (UID: "fc00c131-1994-427e-91dd-b89632fa6720"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:15.123375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.123294    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-tmp-dir" pod "fc00c131-1994-427e-91dd-b89632fa6720" (UID: "fc00c131-1994-427e-91dd-b89632fa6720"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:15.123486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.123468    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-model-cache" pod "fc00c131-1994-427e-91dd-b89632fa6720" (UID: "fc00c131-1994-427e-91dd-b89632fa6720"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:15.124375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.124272    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-dshm" pod "fc00c131-1994-427e-91dd-b89632fa6720" (UID: "fc00c131-1994-427e-91dd-b89632fa6720"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:15.124809 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.124785    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc00c131-1994-427e-91dd-b89632fa6720-tls-certs" pod "fc00c131-1994-427e-91dd-b89632fa6720" (UID: "fc00c131-1994-427e-91dd-b89632fa6720"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:21:15.125830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.125800    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc00c131-1994-427e-91dd-b89632fa6720-kube-api-access-bt9sh" pod "fc00c131-1994-427e-91dd-b89632fa6720" (UID: "fc00c131-1994-427e-91dd-b89632fa6720"). InnerVolumeSpecName "kube-api-access-bt9sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:21:15.224367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.224243    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc00c131-1994-427e-91dd-b89632fa6720-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:15.224367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.224286    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:15.224367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.224301    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:15.224367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.224315    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt9sh\" (UniqueName: \"kubernetes.io/projected/fc00c131-1994-427e-91dd-b89632fa6720-kube-api-access-bt9sh\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:15.224367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.224328    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:15.224367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.224340    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc00c131-1994-427e-91dd-b89632fa6720-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:15.423600 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.423557    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"]
Nov 02 14:21:15.435495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:15.435460    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-2daae4/auth-test-model-2daae4-kserve-8545fc47c-h6pzl"]
Nov 02 14:21:16.170602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:16.170568    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc00c131-1994-427e-91dd-b89632fa6720" path="/var/lib/kubelet/pods/fc00c131-1994-427e-91dd-b89632fa6720/volumes"
Nov 02 14:21:19.506520 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.506451    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:21:19.581783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.581744    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-socket\") pod \"6d853398-09b5-474a-967f-bc2a47e862c3\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") "
Nov 02 14:21:19.582016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.581808    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-credential-socket\") pod \"6d853398-09b5-474a-967f-bc2a47e862c3\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") "
Nov 02 14:21:19.582016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.581836    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-certs\") pod \"6d853398-09b5-474a-967f-bc2a47e862c3\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") "
Nov 02 14:21:19.582016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.581864    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-istio-token\" (UniqueName: \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-istio-token\") pod \"6d853398-09b5-474a-967f-bc2a47e862c3\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") "
Nov 02 14:21:19.582016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.581912    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d853398-09b5-474a-967f-bc2a47e862c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d853398-09b5-474a-967f-bc2a47e862c3-istiod-ca-cert\") pod \"6d853398-09b5-474a-967f-bc2a47e862c3\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") "
Nov 02 14:21:19.582251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582032    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-envoy\") pod \"6d853398-09b5-474a-967f-bc2a47e862c3\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") "
Nov 02 14:21:19.582251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582118    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-kube-api-access-tmf6b\" (UniqueName: \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-kube-api-access-tmf6b\") pod \"6d853398-09b5-474a-967f-bc2a47e862c3\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") "
Nov 02 14:21:19.582251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582158    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-data\") pod \"6d853398-09b5-474a-967f-bc2a47e862c3\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") "
Nov 02 14:21:19.582251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582172    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-socket" pod "6d853398-09b5-474a-967f-bc2a47e862c3" (UID: "6d853398-09b5-474a-967f-bc2a47e862c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:19.582251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582240    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d853398-09b5-474a-967f-bc2a47e862c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d853398-09b5-474a-967f-bc2a47e862c3-istio-podinfo\") pod \"6d853398-09b5-474a-967f-bc2a47e862c3\" (UID: \"6d853398-09b5-474a-967f-bc2a47e862c3\") "
Nov 02 14:21:19.582549 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582524    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-credential-socket" pod "6d853398-09b5-474a-967f-bc2a47e862c3" (UID: "6d853398-09b5-474a-967f-bc2a47e862c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:19.582620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582466    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d853398-09b5-474a-967f-bc2a47e862c3-istiod-ca-cert" pod "6d853398-09b5-474a-967f-bc2a47e862c3" (UID: "6d853398-09b5-474a-967f-bc2a47e862c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:21:19.582764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582699    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-data" pod "6d853398-09b5-474a-967f-bc2a47e862c3" (UID: "6d853398-09b5-474a-967f-bc2a47e862c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:19.582764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582754    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-certs" pod "6d853398-09b5-474a-967f-bc2a47e862c3" (UID: "6d853398-09b5-474a-967f-bc2a47e862c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:19.582896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582794    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:19.582896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582816    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:19.582896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.582830    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d853398-09b5-474a-967f-bc2a47e862c3-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:19.584722 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.584674    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-envoy" pod "6d853398-09b5-474a-967f-bc2a47e862c3" (UID: "6d853398-09b5-474a-967f-bc2a47e862c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:19.584992 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.584922    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d853398-09b5-474a-967f-bc2a47e862c3-istio-podinfo" pod "6d853398-09b5-474a-967f-bc2a47e862c3" (UID: "6d853398-09b5-474a-967f-bc2a47e862c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:21:19.584992 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.584974    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-kube-api-access-tmf6b" pod "6d853398-09b5-474a-967f-bc2a47e862c3" (UID: "6d853398-09b5-474a-967f-bc2a47e862c3"). InnerVolumeSpecName "kube-api-access-tmf6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:21:19.585119 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.585010    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-istio-token" pod "6d853398-09b5-474a-967f-bc2a47e862c3" (UID: "6d853398-09b5-474a-967f-bc2a47e862c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:21:19.685739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.685701    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:19.685739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.685740    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:19.685990 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.685756    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmf6b\" (UniqueName: \"kubernetes.io/projected/6d853398-09b5-474a-967f-bc2a47e862c3-kube-api-access-tmf6b\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:19.685990 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.685771    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:19.685990 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.685784    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d853398-09b5-474a-967f-bc2a47e862c3-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:19.685990 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:19.685797    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d853398-09b5-474a-967f-bc2a47e862c3-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:20.127104 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:20.127069    6284 generic.go:356] "Generic (PLEG): container finished" podID="6d853398-09b5-474a-967f-bc2a47e862c3" containerID="116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3" exitCode=0
Nov 02 14:21:20.127303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:20.127119    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"
Nov 02 14:21:20.127303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:20.127158    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" event={"ID":"6d853398-09b5-474a-967f-bc2a47e862c3","Type":"ContainerDied","Data":"116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3"}
Nov 02 14:21:20.127303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:20.127224    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj" event={"ID":"6d853398-09b5-474a-967f-bc2a47e862c3","Type":"ContainerDied","Data":"44719ecd2e4a23177da4a779a2c048b431b22aefe4a7b3f2d800f74de1f6ee97"}
Nov 02 14:21:20.127303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:20.127253    6284 scope.go:122] "RemoveContainer" containerID="116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3"
Nov 02 14:21:20.138928 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:20.138905    6284 scope.go:122] "RemoveContainer" containerID="116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3"
Nov 02 14:21:20.139313 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:21:20.139293    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3\": container with ID starting with 116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3 not found: ID does not exist" containerID="116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3"
Nov 02 14:21:20.139386 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:20.139330    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3"} err="failed to get container status \"116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3\": rpc error: code = NotFound desc = could not find container \"116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3\": container with ID starting with 116bca61588f9339fdd1c201062478595210cf467fda5defa90a335257587eb3 not found: ID does not exist"
Nov 02 14:21:20.183239 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:20.183206    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"]
Nov 02 14:21:20.192711 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:20.192676    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-nzfzj"]
Nov 02 14:21:20.597661 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:20.597626    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"]
Nov 02 14:21:22.172639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.172601    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" path="/var/lib/kubelet/pods/6d853398-09b5-474a-967f-bc2a47e862c3/volumes"
Nov 02 14:21:22.279863 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.279826    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"]
Nov 02 14:21:22.280444 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280428    6284 cpu_manager.go:414] "containerMap: removing container" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy"
Nov 02 14:21:22.280492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280446    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy"
Nov 02 14:21:22.280492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280461    6284 cpu_manager.go:414] "containerMap: removing container" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" containerName="istio-proxy"
Nov 02 14:21:22.280492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280466    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" containerName="istio-proxy"
Nov 02 14:21:22.280492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280479    6284 cpu_manager.go:414] "containerMap: removing container" podUID="fc00c131-1994-427e-91dd-b89632fa6720" containerName="main"
Nov 02 14:21:22.280492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280486    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc00c131-1994-427e-91dd-b89632fa6720" containerName="main"
Nov 02 14:21:22.280492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280495    6284 cpu_manager.go:414] "containerMap: removing container" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280500    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280514    6284 cpu_manager.go:414] "containerMap: removing container" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280519    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280540    6284 cpu_manager.go:414] "containerMap: removing container" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280545    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280552    6284 cpu_manager.go:414] "containerMap: removing container" podUID="53e527d2-ba69-4c74-8d10-14d8fc00eeed" containerName="main"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280558    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e527d2-ba69-4c74-8d10-14d8fc00eeed" containerName="main"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280564    6284 cpu_manager.go:414] "containerMap: removing container" podUID="10f23f21-4838-46e0-8bd4-03973454b326" containerName="main"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280568    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f23f21-4838-46e0-8bd4-03973454b326" containerName="main"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280654    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e13ec12-9d79-4a64-ba6d-914250c86d2d" containerName="istio-proxy"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280662    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e527d2-ba69-4c74-8d10-14d8fc00eeed" containerName="main"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280669    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f23f21-4838-46e0-8bd4-03973454b326" containerName="main"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280678    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="907e32be-943b-47c5-8091-2bc94b31f433" containerName="istio-proxy"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280684    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d853398-09b5-474a-967f-bc2a47e862c3" containerName="istio-proxy"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280693    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc00c131-1994-427e-91dd-b89632fa6720" containerName="main"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280699    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="982cc2a7-4243-406e-be09-462c0dd2d06a" containerName="istio-proxy"
Nov 02 14:21:22.280754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.280706    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf0c6ad-93b1-4aff-bd98-49aaf2ab01fd" containerName="istio-proxy"
Nov 02 14:21:22.285387 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.285364    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.309751 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.309719    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-5f98a6-serving-cert\""
Nov 02 14:21:22.312993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.312967    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"]
Nov 02 14:21:22.420371 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.420337    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.420392    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef217bd-b6ff-4c32-9591-8584e524bcd5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.420529    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgg9\" (UniqueName: \"kubernetes.io/projected/6ef217bd-b6ff-4c32-9591-8584e524bcd5-kube-api-access-crgg9\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.420642 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.420597    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-metrics-certs\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.522396 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.522357    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef217bd-b6ff-4c32-9591-8584e524bcd5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.522601 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.522445    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgg9\" (UniqueName: \"kubernetes.io/projected/6ef217bd-b6ff-4c32-9591-8584e524bcd5-kube-api-access-crgg9\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.522601 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.522514    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-metrics-certs\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.522722 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.522641    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.523207 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.523177    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef217bd-b6ff-4c32-9591-8584e524bcd5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.525090 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.525067    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-metrics-certs\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.525194 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.525106    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.543640 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.543608    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgg9\" (UniqueName: \"kubernetes.io/projected/6ef217bd-b6ff-4c32-9591-8584e524bcd5-kube-api-access-crgg9\") pod \"maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.600101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.600063    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:22.917395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.917364    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"]
Nov 02 14:21:22.919687 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:21:22.919660    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef217bd_b6ff_4c32_9591_8584e524bcd5.slice/crio-4592fee834004a59108efc86800efaf03996766b6ab28c1cb6f5f6b9357c9d0b WatchSource:0}: Error finding container 4592fee834004a59108efc86800efaf03996766b6ab28c1cb6f5f6b9357c9d0b: Status 404 returned error can't find the container with id 4592fee834004a59108efc86800efaf03996766b6ab28c1cb6f5f6b9357c9d0b
Nov 02 14:21:22.921207 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:22.921183    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:21:23.151015 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:23.150970    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl" event={"ID":"6ef217bd-b6ff-4c32-9591-8584e524bcd5","Type":"ContainerStarted","Data":"4592fee834004a59108efc86800efaf03996766b6ab28c1cb6f5f6b9357c9d0b"}
Nov 02 14:21:24.159484 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:24.159446    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl" event={"ID":"6ef217bd-b6ff-4c32-9591-8584e524bcd5","Type":"ContainerStarted","Data":"7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965"}
Nov 02 14:21:24.160040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:24.159520    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:24.200202 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:24.200141    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl" podStartSLOduration=1.833899357 podStartE2EDuration="2.200123993s" totalImagesPullingTime="366.224636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:24.197242846 +0000 UTC m=+2216.156618454" watchObservedRunningTime="2026-11-02 14:21:24.200123993 +0000 UTC m=+2216.159499597"
Nov 02 14:21:30.172955 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:30.172859    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:21:35.620157 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:35.620101    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs" podUID="058eaa11-e58f-4867-80c0-fc1024b9805c" containerName="main" containerID="cri-o://b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701" gracePeriod=45
Nov 02 14:21:35.653093 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:35.653043    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs" podUID="058eaa11-e58f-4867-80c0-fc1024b9805c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": dial tcp 10.134.0.114:8000: connect: connection refused"
Nov 02 14:21:35.902949 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:35.902901    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:21:36.019327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.019279    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/058eaa11-e58f-4867-80c0-fc1024b9805c-tls-certs\" (UniqueName: \"kubernetes.io/secret/058eaa11-e58f-4867-80c0-fc1024b9805c-tls-certs\") pod \"058eaa11-e58f-4867-80c0-fc1024b9805c\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") "
Nov 02 14:21:36.019327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.019332    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-model-cache\") pod \"058eaa11-e58f-4867-80c0-fc1024b9805c\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") "
Nov 02 14:21:36.019581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.019437    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-tmp-dir\") pod \"058eaa11-e58f-4867-80c0-fc1024b9805c\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") "
Nov 02 14:21:36.019581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.019483    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-dshm\") pod \"058eaa11-e58f-4867-80c0-fc1024b9805c\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") "
Nov 02 14:21:36.019581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.019553    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058eaa11-e58f-4867-80c0-fc1024b9805c-kube-api-access-2bvbn\" (UniqueName: \"kubernetes.io/projected/058eaa11-e58f-4867-80c0-fc1024b9805c-kube-api-access-2bvbn\") pod \"058eaa11-e58f-4867-80c0-fc1024b9805c\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") "
Nov 02 14:21:36.019740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.019620    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-home\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-home\") pod \"058eaa11-e58f-4867-80c0-fc1024b9805c\" (UID: \"058eaa11-e58f-4867-80c0-fc1024b9805c\") "
Nov 02 14:21:36.019740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.019630    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-model-cache" pod "058eaa11-e58f-4867-80c0-fc1024b9805c" (UID: "058eaa11-e58f-4867-80c0-fc1024b9805c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:36.020621 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.020147    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-tmp-dir" pod "058eaa11-e58f-4867-80c0-fc1024b9805c" (UID: "058eaa11-e58f-4867-80c0-fc1024b9805c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:36.020770 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.020695    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:36.020770 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.020717    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:36.021139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.020978    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-home" pod "058eaa11-e58f-4867-80c0-fc1024b9805c" (UID: "058eaa11-e58f-4867-80c0-fc1024b9805c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:36.024498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.024462    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058eaa11-e58f-4867-80c0-fc1024b9805c-tls-certs" pod "058eaa11-e58f-4867-80c0-fc1024b9805c" (UID: "058eaa11-e58f-4867-80c0-fc1024b9805c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:21:36.025595 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.025563    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-dshm" pod "058eaa11-e58f-4867-80c0-fc1024b9805c" (UID: "058eaa11-e58f-4867-80c0-fc1024b9805c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:21:36.025876 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.025851    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058eaa11-e58f-4867-80c0-fc1024b9805c-kube-api-access-2bvbn" pod "058eaa11-e58f-4867-80c0-fc1024b9805c" (UID: "058eaa11-e58f-4867-80c0-fc1024b9805c"). InnerVolumeSpecName "kube-api-access-2bvbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:21:36.121773 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.121726    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:36.121773 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.121766    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bvbn\" (UniqueName: \"kubernetes.io/projected/058eaa11-e58f-4867-80c0-fc1024b9805c-kube-api-access-2bvbn\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:36.121773 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.121778    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/058eaa11-e58f-4867-80c0-fc1024b9805c-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:36.122096 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.121788    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/058eaa11-e58f-4867-80c0-fc1024b9805c-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:21:36.283963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.274165    6284 generic.go:356] "Generic (PLEG): container finished" podID="058eaa11-e58f-4867-80c0-fc1024b9805c" containerID="b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701" exitCode=0
Nov 02 14:21:36.283963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.281982    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"
Nov 02 14:21:36.300798 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.300146    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs" event={"ID":"058eaa11-e58f-4867-80c0-fc1024b9805c","Type":"ContainerDied","Data":"b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701"}
Nov 02 14:21:36.300798 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.300218    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs" event={"ID":"058eaa11-e58f-4867-80c0-fc1024b9805c","Type":"ContainerDied","Data":"c7424a4ea85844db20ac76f306e3daaae05df22375c2a37c4a7b38cc6f87a7e9"}
Nov 02 14:21:36.300798 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.300247    6284 scope.go:122] "RemoveContainer" containerID="b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701"
Nov 02 14:21:36.346956 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.346772    6284 scope.go:122] "RemoveContainer" containerID="b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701"
Nov 02 14:21:36.347224 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:21:36.347195    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701\": container with ID starting with b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701 not found: ID does not exist" containerID="b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701"
Nov 02 14:21:36.347336 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.347239    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701"} err="failed to get container status \"b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701\": rpc error: code = NotFound desc = could not find container \"b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701\": container with ID starting with b6992f95aebde81b098d73db14a433273a5596b21a6c18c2fd7e20a8a5257701 not found: ID does not exist"
Nov 02 14:21:36.366189 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.366149    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"]
Nov 02 14:21:36.371425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:36.371391    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/auth-test-model-21baa7-kserve-b49b8df4f-rqszs"]
Nov 02 14:21:38.170052 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:38.170017    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058eaa11-e58f-4867-80c0-fc1024b9805c" path="/var/lib/kubelet/pods/058eaa11-e58f-4867-80c0-fc1024b9805c/volumes"
Nov 02 14:21:43.896679 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:21:43.896638    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:21:48.774834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.774790    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"]
Nov 02 14:21:48.775642 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.775623    6284 cpu_manager.go:414] "containerMap: removing container" podUID="058eaa11-e58f-4867-80c0-fc1024b9805c" containerName="main"
Nov 02 14:21:48.775740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.775643    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058eaa11-e58f-4867-80c0-fc1024b9805c" containerName="main"
Nov 02 14:21:48.775812 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.775801    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="058eaa11-e58f-4867-80c0-fc1024b9805c" containerName="main"
Nov 02 14:21:48.781218 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.781196    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:48.798997 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.798965    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"]
Nov 02 14:21:48.805031 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.805003    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-87efd3-serving-cert\""
Nov 02 14:21:48.927035 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.926996    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6dp\" (UniqueName: \"kubernetes.io/projected/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-kube-api-access-vq6dp\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:48.927304 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.927120    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-metrics-certs\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:48.927395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.927312    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:48.927475 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:48.927456    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-maas-api-tls\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:49.028928 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:49.028830    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6dp\" (UniqueName: \"kubernetes.io/projected/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-kube-api-access-vq6dp\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:49.028928 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:49.028888    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-metrics-certs\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:49.029241 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:49.029137    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:49.029342 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:49.029317    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-maas-api-tls\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:49.030002 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:49.029924    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:49.032034 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:49.032009    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-metrics-certs\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:49.032152 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:49.032009    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-maas-api-tls\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:49.050316 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:49.050282    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6dp\" (UniqueName: \"kubernetes.io/projected/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-kube-api-access-vq6dp\") pod \"maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:49.100248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:49.100209    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:49.431026 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:49.430998    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"]
Nov 02 14:21:49.433969 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:21:49.433900    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaebe7ed_d010_4e35_a7f5_4a1aaf5e0c50.slice/crio-44afccf4ad9b6d07c6e8440a91829d5412c85c465fe03dc204306c34c9aaf95f WatchSource:0}: Error finding container 44afccf4ad9b6d07c6e8440a91829d5412c85c465fe03dc204306c34c9aaf95f: Status 404 returned error can't find the container with id 44afccf4ad9b6d07c6e8440a91829d5412c85c465fe03dc204306c34c9aaf95f
Nov 02 14:21:50.419840 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:50.419788    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2" event={"ID":"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50","Type":"ContainerStarted","Data":"4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6"}
Nov 02 14:21:50.420350 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:50.419869    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2" event={"ID":"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50","Type":"ContainerStarted","Data":"44afccf4ad9b6d07c6e8440a91829d5412c85c465fe03dc204306c34c9aaf95f"}
Nov 02 14:21:50.420350 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:50.419905    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:50.458689 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:50.458616    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2" podStartSLOduration=2.041981687 podStartE2EDuration="2.458588781s" totalImagesPullingTime="416.607094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:50.4563583 +0000 UTC m=+2242.415733907" watchObservedRunningTime="2026-11-02 14:21:50.458588781 +0000 UTC m=+2242.417964418"
Nov 02 14:21:56.431582 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:56.431506    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:21:59.972020 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:59.971980    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"]
Nov 02 14:21:59.976772 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:59.976747    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:21:59.996355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:21:59.996315    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"]
Nov 02 14:22:00.038983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:00.037024    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2d13f4\""
Nov 02 14:22:00.038983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:00.038162    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2d13f4-dockercfg-lg4dd\""
Nov 02 14:22:00.096970 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:00.096865    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88qk\" (UniqueName: \"kubernetes.io/projected/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-kube-api-access-k88qk\") pod \"payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g\" (UID: \"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:22:00.097202 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:00.097040    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g\" (UID: \"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:22:00.198778 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:00.198741    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88qk\" (UniqueName: \"kubernetes.io/projected/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-kube-api-access-k88qk\") pod \"payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g\" (UID: \"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:22:00.199038 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:00.198873    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g\" (UID: \"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:22:00.199776 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:00.199751    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g\" (UID: \"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:22:00.221609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:00.221576    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88qk\" (UniqueName: \"kubernetes.io/projected/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-kube-api-access-k88qk\") pod \"payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g\" (UID: \"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:22:00.293131 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:00.293084    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:22:00.601761 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:00.601615    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"]
Nov 02 14:22:00.605091 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:22:00.605053    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2d075c_f2bb_4aeb_a32a_ef78bfae41c4.slice/crio-cb129efeb7a596f7a4c150e7e74df1c3b4956d72f2066ce6673dd26fb1d5a997 WatchSource:0}: Error finding container cb129efeb7a596f7a4c150e7e74df1c3b4956d72f2066ce6673dd26fb1d5a997: Status 404 returned error can't find the container with id cb129efeb7a596f7a4c150e7e74df1c3b4956d72f2066ce6673dd26fb1d5a997
Nov 02 14:22:01.496930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:01.496888    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g" event={"ID":"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4","Type":"ContainerStarted","Data":"69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb"}
Nov 02 14:22:01.497512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:01.497026    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:22:01.497512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:01.497052    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g" event={"ID":"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4","Type":"ContainerStarted","Data":"cb129efeb7a596f7a4c150e7e74df1c3b4956d72f2066ce6673dd26fb1d5a997"}
Nov 02 14:22:01.547736 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:01.547667    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g" podStartSLOduration=2.5476478 podStartE2EDuration="2.5476478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:01.54371946 +0000 UTC m=+2253.503095065" watchObservedRunningTime="2026-11-02 14:22:01.5476478 +0000 UTC m=+2253.507023406"
Nov 02 14:22:08.662200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.662154    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"]
Nov 02 14:22:08.666766 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.666736    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.693111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.693077    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-0c2554-serving-cert\""
Nov 02 14:22:08.699164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.699121    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"]
Nov 02 14:22:08.820713 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.820665    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66eff972-2f71-45a8-af06-b5db71a265d0-openshift-service-ca\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.820951 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.820821    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-metrics-certs\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.820951 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.820887    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-maas-api-tls\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.821047 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.820981    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bpk\" (UniqueName: \"kubernetes.io/projected/66eff972-2f71-45a8-af06-b5db71a265d0-kube-api-access-q2bpk\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.922465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.922367    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-metrics-certs\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.922465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.922447    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-maas-api-tls\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.922727 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.922486    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bpk\" (UniqueName: \"kubernetes.io/projected/66eff972-2f71-45a8-af06-b5db71a265d0-kube-api-access-q2bpk\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.922727 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:22:08.922605    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-serving-cert: secret "maas-api-e2e-autores-0c2554-serving-cert" not found
Nov 02 14:22:08.922727 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.922629    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66eff972-2f71-45a8-af06-b5db71a265d0-openshift-service-ca\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.922727 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:22:08.922684    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-maas-api-tls podName:66eff972-2f71-45a8-af06-b5db71a265d0 nodeName:}" failed. No retries permitted until 2026-11-02 14:22:09.422660431 +0000 UTC m=+2261.382036033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-maas-api-tls") pod "maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh" (UID: "66eff972-2f71-45a8-af06-b5db71a265d0") : secret "maas-api-e2e-autores-0c2554-serving-cert" not found
Nov 02 14:22:08.923570 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.923536    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66eff972-2f71-45a8-af06-b5db71a265d0-openshift-service-ca\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.925272 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.925246    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-metrics-certs\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:08.944502 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:08.944468    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bpk\" (UniqueName: \"kubernetes.io/projected/66eff972-2f71-45a8-af06-b5db71a265d0-kube-api-access-q2bpk\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:09.429240 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:09.429202    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-maas-api-tls\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:09.431700 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:09.431675    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-maas-api-tls\") pod \"maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:09.586927 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:09.586876    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:09.931575 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:09.931523    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"]
Nov 02 14:22:09.940267 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:22:09.940231    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66eff972_2f71_45a8_af06_b5db71a265d0.slice/crio-85701f5b7adb90e6460fd992d3841059f10cc36549b2433dcdaa2977f2f09fb1 WatchSource:0}: Error finding container 85701f5b7adb90e6460fd992d3841059f10cc36549b2433dcdaa2977f2f09fb1: Status 404 returned error can't find the container with id 85701f5b7adb90e6460fd992d3841059f10cc36549b2433dcdaa2977f2f09fb1
Nov 02 14:22:10.571787 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:10.571743    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh" event={"ID":"66eff972-2f71-45a8-af06-b5db71a265d0","Type":"ContainerStarted","Data":"f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87"}
Nov 02 14:22:10.572025 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:10.571823    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:10.572025 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:10.571850    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh" event={"ID":"66eff972-2f71-45a8-af06-b5db71a265d0","Type":"ContainerStarted","Data":"85701f5b7adb90e6460fd992d3841059f10cc36549b2433dcdaa2977f2f09fb1"}
Nov 02 14:22:10.616816 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:10.616584    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh" podStartSLOduration=2.160964432 podStartE2EDuration="2.616564474s" totalImagesPullingTime="455.600042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:10.615712174 +0000 UTC m=+2262.575087784" watchObservedRunningTime="2026-11-02 14:22:10.616564474 +0000 UTC m=+2262.575940082"
Nov 02 14:22:12.519412 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:12.519378    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:22:13.909050 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:22:13.909009    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:22:16.584667 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:16.584633    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:22:21.333609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.333574    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"]
Nov 02 14:22:21.338178 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.338155    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.350352 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.350325    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-ee3eca-serving-cert\""
Nov 02 14:22:21.368461 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.368425    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"]
Nov 02 14:22:21.446837 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.446788    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-metrics-certs\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.447097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.446860    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.447097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.447047    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262e5c36-6101-405c-899e-78712757602c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.447289 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.447172    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2q7x\" (UniqueName: \"kubernetes.io/projected/262e5c36-6101-405c-899e-78712757602c-kube-api-access-t2q7x\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.548512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.548469    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-metrics-certs\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.548814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.548539    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.548814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.548581    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262e5c36-6101-405c-899e-78712757602c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.548814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.548633    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2q7x\" (UniqueName: \"kubernetes.io/projected/262e5c36-6101-405c-899e-78712757602c-kube-api-access-t2q7x\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.549496 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.549455    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262e5c36-6101-405c-899e-78712757602c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.551987 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.551921    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-metrics-certs\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.551987 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.551921    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.570111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.570065    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2q7x\" (UniqueName: \"kubernetes.io/projected/262e5c36-6101-405c-899e-78712757602c-kube-api-access-t2q7x\") pod \"maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.653242 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.653149    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:21.976966 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:21.976918    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"]
Nov 02 14:22:21.979999 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:22:21.979949    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262e5c36_6101_405c_899e_78712757602c.slice/crio-9e9a398749134ba785c28ae735b9f021accaac6b5fbdcbcbaef43965c10d5f83 WatchSource:0}: Error finding container 9e9a398749134ba785c28ae735b9f021accaac6b5fbdcbcbaef43965c10d5f83: Status 404 returned error can't find the container with id 9e9a398749134ba785c28ae735b9f021accaac6b5fbdcbcbaef43965c10d5f83
Nov 02 14:22:22.733064 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:22.732970    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls" event={"ID":"262e5c36-6101-405c-899e-78712757602c","Type":"ContainerStarted","Data":"f3f55d629e28469eee57fb404d8ba7bbc0dc6c419cb6e1ed1472bff7fc074849"}
Nov 02 14:22:22.733064 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:22.733030    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls" event={"ID":"262e5c36-6101-405c-899e-78712757602c","Type":"ContainerStarted","Data":"9e9a398749134ba785c28ae735b9f021accaac6b5fbdcbcbaef43965c10d5f83"}
Nov 02 14:22:22.733491 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:22.733199    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:22.777695 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:22.777633    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls" podStartSLOduration=1.425463814 podStartE2EDuration="1.777616547s" totalImagesPullingTime="352.152733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:22.773680873 +0000 UTC m=+2274.733056478" watchObservedRunningTime="2026-11-02 14:22:22.777616547 +0000 UTC m=+2274.736992207"
Nov 02 14:22:28.746798 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:28.746767    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:22:35.388850 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.388804    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"]
Nov 02 14:22:35.393863 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.393829    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.408388 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.408357    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2d13f4-serving-cert\""
Nov 02 14:22:35.422628 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.422570    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"]
Nov 02 14:22:35.457598 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.457564    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.457780 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.457615    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5515bb77-f206-4c6e-9f0f-3f800a408723-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.457844 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.457816    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lpz\" (UniqueName: \"kubernetes.io/projected/5515bb77-f206-4c6e-9f0f-3f800a408723-kube-api-access-b5lpz\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.457897 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.457870    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-metrics-certs\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.560958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.559231    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lpz\" (UniqueName: \"kubernetes.io/projected/5515bb77-f206-4c6e-9f0f-3f800a408723-kube-api-access-b5lpz\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.560958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.559301    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-metrics-certs\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.560958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.559496    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.560958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.559547    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5515bb77-f206-4c6e-9f0f-3f800a408723-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.560958 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:22:35.560283    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-serving-cert: secret "maas-api-e2e-extmod-a-2d13f4-serving-cert" not found
Nov 02 14:22:35.560958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.560659    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5515bb77-f206-4c6e-9f0f-3f800a408723-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.560958 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:22:35.560690    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-maas-api-tls podName:5515bb77-f206-4c6e-9f0f-3f800a408723 nodeName:}" failed. No retries permitted until 2026-11-02 14:22:36.060664968 +0000 UTC m=+2288.020040567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-maas-api-tls") pod "maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf" (UID: "5515bb77-f206-4c6e-9f0f-3f800a408723") : secret "maas-api-e2e-extmod-a-2d13f4-serving-cert" not found
Nov 02 14:22:35.562919 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.562892    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-metrics-certs\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:35.580975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:35.580894    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lpz\" (UniqueName: \"kubernetes.io/projected/5515bb77-f206-4c6e-9f0f-3f800a408723-kube-api-access-b5lpz\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:36.066150 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:36.066112    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:36.069105 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:36.069070    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:36.312111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:36.312071    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:36.644053 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:36.644018    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"]
Nov 02 14:22:36.650134 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:22:36.650098    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5515bb77_f206_4c6e_9f0f_3f800a408723.slice/crio-8c9835a98ea232bf110b8b685968d0d0bf145872f6e1567d24a4fce3d718309b WatchSource:0}: Error finding container 8c9835a98ea232bf110b8b685968d0d0bf145872f6e1567d24a4fce3d718309b: Status 404 returned error can't find the container with id 8c9835a98ea232bf110b8b685968d0d0bf145872f6e1567d24a4fce3d718309b
Nov 02 14:22:36.841298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:36.841259    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf" event={"ID":"5515bb77-f206-4c6e-9f0f-3f800a408723","Type":"ContainerStarted","Data":"8c9835a98ea232bf110b8b685968d0d0bf145872f6e1567d24a4fce3d718309b"}
Nov 02 14:22:37.850789 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:37.850745    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf" event={"ID":"5515bb77-f206-4c6e-9f0f-3f800a408723","Type":"ContainerStarted","Data":"beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126"}
Nov 02 14:22:37.851294 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:37.850848    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:37.889154 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:37.889085    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf" podStartSLOduration=2.543216893 podStartE2EDuration="2.889066126s" totalImagesPullingTime="345.849233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:37.8883835 +0000 UTC m=+2289.847759108" watchObservedRunningTime="2026-11-02 14:22:37.889066126 +0000 UTC m=+2289.848441734"
Nov 02 14:22:43.863166 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:43.863132    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:22:43.896359 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:22:43.896321    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:22:49.279981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.279915    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"]
Nov 02 14:22:49.287254 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.287205    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.324539 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.324502    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a02c35-serving-cert\""
Nov 02 14:22:49.330683 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.330641    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"]
Nov 02 14:22:49.466783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.466742    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-metrics-certs\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.467048 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.466902    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.467048 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.467023    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/274d4228-83fc-4efb-962b-63f1dbf48ac8-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.467163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.467134    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgsq2\" (UniqueName: \"kubernetes.io/projected/274d4228-83fc-4efb-962b-63f1dbf48ac8-kube-api-access-vgsq2\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.572209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.572109    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgsq2\" (UniqueName: \"kubernetes.io/projected/274d4228-83fc-4efb-962b-63f1dbf48ac8-kube-api-access-vgsq2\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.572209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.572194    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-metrics-certs\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.572407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.572292    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.572407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.572354    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/274d4228-83fc-4efb-962b-63f1dbf48ac8-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.573551 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.573256    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/274d4228-83fc-4efb-962b-63f1dbf48ac8-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.577012 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.576102    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.579979 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.578387    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-metrics-certs\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.609808 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.609778    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgsq2\" (UniqueName: \"kubernetes.io/projected/274d4228-83fc-4efb-962b-63f1dbf48ac8-kube-api-access-vgsq2\") pod \"maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:49.902275 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:49.902168    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:50.221342 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:50.221307    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"]
Nov 02 14:22:50.222483 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:22:50.222454    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274d4228_83fc_4efb_962b_63f1dbf48ac8.slice/crio-34bda03f9f34780a1a7350f346747166f4979f4df7a90ad65ec1519340eb0226 WatchSource:0}: Error finding container 34bda03f9f34780a1a7350f346747166f4979f4df7a90ad65ec1519340eb0226: Status 404 returned error can't find the container with id 34bda03f9f34780a1a7350f346747166f4979f4df7a90ad65ec1519340eb0226
Nov 02 14:22:50.943259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:50.943163    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw" event={"ID":"274d4228-83fc-4efb-962b-63f1dbf48ac8","Type":"ContainerStarted","Data":"6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1"}
Nov 02 14:22:50.943259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:50.943228    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw" event={"ID":"274d4228-83fc-4efb-962b-63f1dbf48ac8","Type":"ContainerStarted","Data":"34bda03f9f34780a1a7350f346747166f4979f4df7a90ad65ec1519340eb0226"}
Nov 02 14:22:50.943259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:50.943261    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:22:50.984524 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:50.984466    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw" podStartSLOduration=1.661113673 podStartE2EDuration="1.984450285s" totalImagesPullingTime="323.336612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:50.980688402 +0000 UTC m=+2302.940064021" watchObservedRunningTime="2026-11-02 14:22:50.984450285 +0000 UTC m=+2302.943825895"
Nov 02 14:22:52.178584 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.178545    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"]
Nov 02 14:22:52.182985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.182955    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.209427 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.209395    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0c2554\"/\"autores-model-0c2554-kserve-self-signed-certs\""
Nov 02 14:22:52.209622 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.209520    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0c2554\"/\"default-dockercfg-47rp8\""
Nov 02 14:22:52.213318 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.213283    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"]
Nov 02 14:22:52.218869 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.218833    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0c2554\"/\"openshift-service-ca.crt\""
Nov 02 14:22:52.219163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.218848    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0c2554\"/\"kube-root-ca.crt\""
Nov 02 14:22:52.222689 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.222651    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tls-certs\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.222884 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.222717    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-home\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.222884 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.222766    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-dshm\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.222884 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.222791    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tmp-dir\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.222884 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.222818    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm85h\" (UniqueName: \"kubernetes.io/projected/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-kube-api-access-qm85h\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.223135 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.222993    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-model-cache\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.324355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.324313    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tls-certs\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.324585 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.324380    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-home\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.324585 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.324432    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-dshm\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.324585 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.324454    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tmp-dir\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.324585 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.324478    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm85h\" (UniqueName: \"kubernetes.io/projected/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-kube-api-access-qm85h\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.324805 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.324642    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-model-cache\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.324976 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.324909    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-home\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.325074 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.325055    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tmp-dir\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.325180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.325160    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-model-cache\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.327194 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.327163    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-dshm\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.327589 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.327567    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tls-certs\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.346055 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.346019    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm85h\" (UniqueName: \"kubernetes.io/projected/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-kube-api-access-qm85h\") pod \"autores-model-0c2554-kserve-bdbbddc77-fwqjf\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") " pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.508022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.507980    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:52.860889 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.860767    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"]
Nov 02 14:22:52.867487 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:22:52.867447    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8caa0fb6_d69d_4bc9_ae0e_f40ee197aab9.slice/crio-c1371e0870adb76ca5a12f76c7d4fae5ab1a0caa0fdabd4f91aed022c89dc367 WatchSource:0}: Error finding container c1371e0870adb76ca5a12f76c7d4fae5ab1a0caa0fdabd4f91aed022c89dc367: Status 404 returned error can't find the container with id c1371e0870adb76ca5a12f76c7d4fae5ab1a0caa0fdabd4f91aed022c89dc367
Nov 02 14:22:52.962667 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.962554    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf" event={"ID":"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9","Type":"ContainerStarted","Data":"47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a"}
Nov 02 14:22:52.962667 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:52.962611    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf" event={"ID":"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9","Type":"ContainerStarted","Data":"c1371e0870adb76ca5a12f76c7d4fae5ab1a0caa0fdabd4f91aed022c89dc367"}
Nov 02 14:22:53.009406 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:53.009351    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf" podStartSLOduration=1.009333757 podStartE2EDuration="1.009333757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:53.004746554 +0000 UTC m=+2304.964122164" watchObservedRunningTime="2026-11-02 14:22:53.009333757 +0000 UTC m=+2304.968709363"
Nov 02 14:22:53.508863 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:53.508819    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:22:56.958187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:22:56.958157    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:23:02.508572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:02.508540    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:23:02.522063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:02.522010    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:23:03.048479 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:03.048448    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:23:13.896532 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:23:13.896488    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:23:17.709833 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.709763    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"]
Nov 02 14:23:17.715141 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.715110    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.727374 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.727338    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-bc34db-serving-cert\""
Nov 02 14:23:17.744135 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.744099    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"]
Nov 02 14:23:17.802478 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.802426    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.802758 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.802739    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-metrics-certs\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.802922 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.802909    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmnzb\" (UniqueName: \"kubernetes.io/projected/da26804e-7a73-49b0-8610-95f68c2e7845-kube-api-access-vmnzb\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.803110 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.803097    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da26804e-7a73-49b0-8610-95f68c2e7845-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.909102 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.909060    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.909403 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.909369    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-metrics-certs\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.909541 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.909479    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmnzb\" (UniqueName: \"kubernetes.io/projected/da26804e-7a73-49b0-8610-95f68c2e7845-kube-api-access-vmnzb\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.909608 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.909568    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da26804e-7a73-49b0-8610-95f68c2e7845-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.910494 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:23:17.910465    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-serving-cert: secret "maas-api-e2e-extmod-b-bc34db-serving-cert" not found
Nov 02 14:23:17.910724 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:23:17.910706    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-maas-api-tls podName:da26804e-7a73-49b0-8610-95f68c2e7845 nodeName:}" failed. No retries permitted until 2026-11-02 14:23:18.410682756 +0000 UTC m=+2330.370058358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-maas-api-tls") pod "maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr" (UID: "da26804e-7a73-49b0-8610-95f68c2e7845") : secret "maas-api-e2e-extmod-b-bc34db-serving-cert" not found
Nov 02 14:23:17.910847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.910710    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da26804e-7a73-49b0-8610-95f68c2e7845-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.913291 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.913120    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-metrics-certs\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:17.932856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:17.932821    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmnzb\" (UniqueName: \"kubernetes.io/projected/da26804e-7a73-49b0-8610-95f68c2e7845-kube-api-access-vmnzb\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:18.416881 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:18.416837    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:18.419501 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:18.419468    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-maas-api-tls\") pod \"maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:18.634277 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:18.634242    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:18.946814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:18.946785    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"]
Nov 02 14:23:18.949396 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:23:18.949363    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda26804e_7a73_49b0_8610_95f68c2e7845.slice/crio-1af7c16a42d443e6754958188db77b7d15a2e0e6381b0bb8a2d7b69340db346c WatchSource:0}: Error finding container 1af7c16a42d443e6754958188db77b7d15a2e0e6381b0bb8a2d7b69340db346c: Status 404 returned error can't find the container with id 1af7c16a42d443e6754958188db77b7d15a2e0e6381b0bb8a2d7b69340db346c
Nov 02 14:23:19.171452 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:19.171414    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr" event={"ID":"da26804e-7a73-49b0-8610-95f68c2e7845","Type":"ContainerStarted","Data":"1af7c16a42d443e6754958188db77b7d15a2e0e6381b0bb8a2d7b69340db346c"}
Nov 02 14:23:19.556728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:19.556629    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"]
Nov 02 14:23:20.179294 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:20.179247    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr" event={"ID":"da26804e-7a73-49b0-8610-95f68c2e7845","Type":"ContainerStarted","Data":"cc00bfa63aa3f3797eb3e03c47c97c0a77b84445923f1e973691628db26826cc"}
Nov 02 14:23:20.179797 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:20.179342    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:20.222085 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:20.222033    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr" podStartSLOduration=2.924464068 podStartE2EDuration="3.222017163s" totalImagesPullingTime="297.553095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:20.218128237 +0000 UTC m=+2332.177503845" watchObservedRunningTime="2026-11-02 14:23:20.222017163 +0000 UTC m=+2332.181392788"
Nov 02 14:23:26.192737 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:26.192657    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:23:34.609838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.609788    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf" podUID="8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" containerName="main" containerID="cri-o://47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a" gracePeriod=45
Nov 02 14:23:34.920219 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.920063    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:23:34.965749 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.965699    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tls-certs\" (UniqueName: \"kubernetes.io/secret/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tls-certs\") pod \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") "
Nov 02 14:23:34.965999 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.965764    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-kube-api-access-qm85h\" (UniqueName: \"kubernetes.io/projected/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-kube-api-access-qm85h\") pod \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") "
Nov 02 14:23:34.965999 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.965882    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-dshm\") pod \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") "
Nov 02 14:23:34.966132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.966017    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-home\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-home\") pod \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") "
Nov 02 14:23:34.966132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.966056    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tmp-dir\") pod \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") "
Nov 02 14:23:34.966132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.966092    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-model-cache\") pod \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\" (UID: \"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9\") "
Nov 02 14:23:34.966869 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.966804    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tmp-dir" pod "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" (UID: "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:34.967114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.966944    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-model-cache" pod "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" (UID: "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:34.967423 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.967390    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-home" pod "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" (UID: "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:34.969299 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.969267    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-dshm" pod "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" (UID: "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:34.970870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.970509    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-kube-api-access-qm85h" pod "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" (UID: "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9"). InnerVolumeSpecName "kube-api-access-qm85h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:34.971043 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:34.970923    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tls-certs" pod "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" (UID: "8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:23:35.067634 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.067591    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:35.067634 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.067625    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:35.067634 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.067638    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:35.067982 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.067653    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:35.067982 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.067665    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:35.067982 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.067676    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm85h\" (UniqueName: \"kubernetes.io/projected/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9-kube-api-access-qm85h\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:35.316080 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.316028    6284 generic.go:356] "Generic (PLEG): container finished" podID="8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" containerID="47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a" exitCode=0
Nov 02 14:23:35.316080 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.316083    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"
Nov 02 14:23:35.316348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.316122    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf" event={"ID":"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9","Type":"ContainerDied","Data":"47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a"}
Nov 02 14:23:35.316348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.316181    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf" event={"ID":"8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9","Type":"ContainerDied","Data":"c1371e0870adb76ca5a12f76c7d4fae5ab1a0caa0fdabd4f91aed022c89dc367"}
Nov 02 14:23:35.316348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.316204    6284 scope.go:122] "RemoveContainer" containerID="47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a"
Nov 02 14:23:35.328086 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.328061    6284 scope.go:122] "RemoveContainer" containerID="47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a"
Nov 02 14:23:35.328489 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:23:35.328464    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a\": container with ID starting with 47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a not found: ID does not exist" containerID="47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a"
Nov 02 14:23:35.328538 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.328503    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a"} err="failed to get container status \"47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a\": rpc error: code = NotFound desc = could not find container \"47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a\": container with ID starting with 47936b9056bc4130cf323075ed78f9763492d6154f2022618c61f4db9f64962a not found: ID does not exist"
Nov 02 14:23:35.368613 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.368566    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"]
Nov 02 14:23:35.373519 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:35.373481    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-0c2554/autores-model-0c2554-kserve-bdbbddc77-fwqjf"]
Nov 02 14:23:36.170478 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:36.170442    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" path="/var/lib/kubelet/pods/8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9/volumes"
Nov 02 14:23:43.896618 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:23:43.896577    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:23:44.635800 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:44.635761    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"]
Nov 02 14:23:44.636083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:44.636047    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerName="istio-proxy" containerID="cri-o://6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51" gracePeriod=30
Nov 02 14:23:48.552214 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:48.552177    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"]
Nov 02 14:23:48.552721 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:48.552386    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy" containerID="cri-o://069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c" gracePeriod=30
Nov 02 14:23:49.921757 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:49.921683    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:23:50.026144 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026108    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-token\" (UniqueName: \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-token\") pod \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") "
Nov 02 14:23:50.026298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026193    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-socket\") pod \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") "
Nov 02 14:23:50.026298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026246    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-certs\") pod \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") "
Nov 02 14:23:50.026298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026269    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-envoy\") pod \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") "
Nov 02 14:23:50.026445 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026381    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-podinfo\") pod \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") "
Nov 02 14:23:50.026504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026479    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-kube-api-access-2t9qv\" (UniqueName: \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-kube-api-access-2t9qv\") pod \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") "
Nov 02 14:23:50.026562 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026514    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istiod-ca-cert\") pod \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") "
Nov 02 14:23:50.026562 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026530    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-certs" pod "c24249a3-c205-40cf-b9d4-87dc53b97ff8" (UID: "c24249a3-c205-40cf-b9d4-87dc53b97ff8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:50.026562 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026548    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-credential-socket\") pod \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") "
Nov 02 14:23:50.026729 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026592    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-data\") pod \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\" (UID: \"c24249a3-c205-40cf-b9d4-87dc53b97ff8\") "
Nov 02 14:23:50.026887 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.026863    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istiod-ca-cert" pod "c24249a3-c205-40cf-b9d4-87dc53b97ff8" (UID: "c24249a3-c205-40cf-b9d4-87dc53b97ff8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:23:50.027206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.027106    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-socket" pod "c24249a3-c205-40cf-b9d4-87dc53b97ff8" (UID: "c24249a3-c205-40cf-b9d4-87dc53b97ff8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:50.027206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.027121    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-credential-socket" pod "c24249a3-c205-40cf-b9d4-87dc53b97ff8" (UID: "c24249a3-c205-40cf-b9d4-87dc53b97ff8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:50.027206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.027187    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:50.027206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.027208    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:50.027482 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.027219    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-data" pod "c24249a3-c205-40cf-b9d4-87dc53b97ff8" (UID: "c24249a3-c205-40cf-b9d4-87dc53b97ff8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:50.028522 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.028498    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-token" pod "c24249a3-c205-40cf-b9d4-87dc53b97ff8" (UID: "c24249a3-c205-40cf-b9d4-87dc53b97ff8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:50.028599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.028551    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-podinfo" pod "c24249a3-c205-40cf-b9d4-87dc53b97ff8" (UID: "c24249a3-c205-40cf-b9d4-87dc53b97ff8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:23:50.028700 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.028680    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-kube-api-access-2t9qv" pod "c24249a3-c205-40cf-b9d4-87dc53b97ff8" (UID: "c24249a3-c205-40cf-b9d4-87dc53b97ff8"). InnerVolumeSpecName "kube-api-access-2t9qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:50.028741 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.028713    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-envoy" pod "c24249a3-c205-40cf-b9d4-87dc53b97ff8" (UID: "c24249a3-c205-40cf-b9d4-87dc53b97ff8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:50.128039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.128001    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:50.128039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.128033    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:50.128039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.128046    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:50.128306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.128055    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t9qv\" (UniqueName: \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-kube-api-access-2t9qv\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:50.128306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.128065    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:50.128306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.128073    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:50.128306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.128082    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24249a3-c205-40cf-b9d4-87dc53b97ff8-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:50.207748 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:23:50.207707    6284 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24249a3_c205_40cf_b9d4_87dc53b97ff8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24249a3_c205_40cf_b9d4_87dc53b97ff8.slice/crio-4bb893afe940fb24b91bbd2af97e281c2fab2276307c5244684da3fc75fc0c5e\": RecentStats: unable to find data in memory cache]"
Nov 02 14:23:50.445012 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.438400    6284 generic.go:356] "Generic (PLEG): container finished" podID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerID="6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51" exitCode=0
Nov 02 14:23:50.445012 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.438528    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" event={"ID":"c24249a3-c205-40cf-b9d4-87dc53b97ff8","Type":"ContainerDied","Data":"6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51"}
Nov 02 14:23:50.445012 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.438589    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn" event={"ID":"c24249a3-c205-40cf-b9d4-87dc53b97ff8","Type":"ContainerDied","Data":"4bb893afe940fb24b91bbd2af97e281c2fab2276307c5244684da3fc75fc0c5e"}
Nov 02 14:23:50.445012 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.438622    6284 scope.go:122] "RemoveContainer" containerID="6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51"
Nov 02 14:23:50.445012 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.439000    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"
Nov 02 14:23:50.456749 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.456720    6284 scope.go:122] "RemoveContainer" containerID="6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51"
Nov 02 14:23:50.457176 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:23:50.457154    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51\": container with ID starting with 6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51 not found: ID does not exist" containerID="6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51"
Nov 02 14:23:50.457261 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.457187    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51"} err="failed to get container status \"6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51\": rpc error: code = NotFound desc = could not find container \"6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51\": container with ID starting with 6bb6ab2bc0b2754e641b7298e251843e8c86a4bcf9c77f41b842a0f0395bfc51 not found: ID does not exist"
Nov 02 14:23:50.489278 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.489240    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"]
Nov 02 14:23:50.501816 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:50.501782    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-ndmzn"]
Nov 02 14:23:52.170656 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:52.170622    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" path="/var/lib/kubelet/pods/c24249a3-c205-40cf-b9d4-87dc53b97ff8/volumes"
Nov 02 14:23:53.830468 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.830404    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:23:53.877718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.877675    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-certs\") pod \"e3d196b8-eeb0-4933-b320-2682b74cb13c\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") "
Nov 02 14:23:53.877914 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.877734    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-token\" (UniqueName: \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-token\") pod \"e3d196b8-eeb0-4933-b320-2682b74cb13c\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") "
Nov 02 14:23:53.877914 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.877822    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-envoy\") pod \"e3d196b8-eeb0-4933-b320-2682b74cb13c\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") "
Nov 02 14:23:53.877914 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.877856    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-kube-api-access-nzpcx\" (UniqueName: \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-kube-api-access-nzpcx\") pod \"e3d196b8-eeb0-4933-b320-2682b74cb13c\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") "
Nov 02 14:23:53.877914 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.877885    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-socket\") pod \"e3d196b8-eeb0-4933-b320-2682b74cb13c\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") "
Nov 02 14:23:53.877914 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.877911    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-podinfo\") pod \"e3d196b8-eeb0-4933-b320-2682b74cb13c\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") "
Nov 02 14:23:53.878218 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.877984    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d196b8-eeb0-4933-b320-2682b74cb13c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d196b8-eeb0-4933-b320-2682b74cb13c-istiod-ca-cert\") pod \"e3d196b8-eeb0-4933-b320-2682b74cb13c\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") "
Nov 02 14:23:53.878218 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878049    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-data\") pod \"e3d196b8-eeb0-4933-b320-2682b74cb13c\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") "
Nov 02 14:23:53.878218 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878096    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-credential-socket\") pod \"e3d196b8-eeb0-4933-b320-2682b74cb13c\" (UID: \"e3d196b8-eeb0-4933-b320-2682b74cb13c\") "
Nov 02 14:23:53.878382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878042    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-certs" pod "e3d196b8-eeb0-4933-b320-2682b74cb13c" (UID: "e3d196b8-eeb0-4933-b320-2682b74cb13c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:53.878382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878262    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-socket" pod "e3d196b8-eeb0-4933-b320-2682b74cb13c" (UID: "e3d196b8-eeb0-4933-b320-2682b74cb13c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:53.878382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878367    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-data" pod "e3d196b8-eeb0-4933-b320-2682b74cb13c" (UID: "e3d196b8-eeb0-4933-b320-2682b74cb13c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:53.878557 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878470    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-credential-socket" pod "e3d196b8-eeb0-4933-b320-2682b74cb13c" (UID: "e3d196b8-eeb0-4933-b320-2682b74cb13c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:53.878557 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878544    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d196b8-eeb0-4933-b320-2682b74cb13c-istiod-ca-cert" pod "e3d196b8-eeb0-4933-b320-2682b74cb13c" (UID: "e3d196b8-eeb0-4933-b320-2682b74cb13c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:23:53.878761 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878740    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d196b8-eeb0-4933-b320-2682b74cb13c-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:53.878841 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878765    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:53.878841 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878778    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:53.878841 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878789    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:53.878841 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.878797    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:53.880533 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.880486    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-kube-api-access-nzpcx" pod "e3d196b8-eeb0-4933-b320-2682b74cb13c" (UID: "e3d196b8-eeb0-4933-b320-2682b74cb13c"). InnerVolumeSpecName "kube-api-access-nzpcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:53.880671 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.880552    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-envoy" pod "e3d196b8-eeb0-4933-b320-2682b74cb13c" (UID: "e3d196b8-eeb0-4933-b320-2682b74cb13c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:53.880671 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.880555    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-token" pod "e3d196b8-eeb0-4933-b320-2682b74cb13c" (UID: "e3d196b8-eeb0-4933-b320-2682b74cb13c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:53.881190 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.881159    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-podinfo" pod "e3d196b8-eeb0-4933-b320-2682b74cb13c" (UID: "e3d196b8-eeb0-4933-b320-2682b74cb13c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:23:53.979784 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.979735    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:53.979784 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.979773    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:53.979784 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.979784    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzpcx\" (UniqueName: \"kubernetes.io/projected/e3d196b8-eeb0-4933-b320-2682b74cb13c-kube-api-access-nzpcx\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:53.979784 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:53.979794    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d196b8-eeb0-4933-b320-2682b74cb13c-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:23:54.469620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:54.469519    6284 generic.go:356] "Generic (PLEG): container finished" podID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerID="069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c" exitCode=0
Nov 02 14:23:54.469620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:54.469571    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"
Nov 02 14:23:54.469620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:54.469601    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" event={"ID":"e3d196b8-eeb0-4933-b320-2682b74cb13c","Type":"ContainerDied","Data":"069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c"}
Nov 02 14:23:54.469977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:54.469667    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2" event={"ID":"e3d196b8-eeb0-4933-b320-2682b74cb13c","Type":"ContainerDied","Data":"a2bd3c5e04f52adcd075f54893c9bae2b66229441a9a953744218916fe69cdfb"}
Nov 02 14:23:54.469977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:54.469692    6284 scope.go:122] "RemoveContainer" containerID="069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c"
Nov 02 14:23:54.481014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:54.480989    6284 scope.go:122] "RemoveContainer" containerID="069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c"
Nov 02 14:23:54.481397 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:23:54.481377    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c\": container with ID starting with 069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c not found: ID does not exist" containerID="069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c"
Nov 02 14:23:54.481450 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:54.481410    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c"} err="failed to get container status \"069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c\": rpc error: code = NotFound desc = could not find container \"069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c\": container with ID starting with 069a1e0fef2b545ab2d8b94ecc1d0dda1ea5a8dd6d029aa76fc3f6ffc4c1579c not found: ID does not exist"
Nov 02 14:23:54.516078 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:54.516029    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"]
Nov 02 14:23:54.525876 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:54.525589    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-l95s2"]
Nov 02 14:23:56.201975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:23:56.186634    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" path="/var/lib/kubelet/pods/e3d196b8-eeb0-4933-b320-2682b74cb13c/volumes"
Nov 02 14:24:12.705547 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:12.705503    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"]
Nov 02 14:24:12.706302 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:12.705864    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh" podUID="66eff972-2f71-45a8-af06-b5db71a265d0" containerName="maas-api" containerID="cri-o://f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87" gracePeriod=30
Nov 02 14:24:13.073367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.073336    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:24:13.160985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.160029    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66eff972-2f71-45a8-af06-b5db71a265d0-kube-api-access-q2bpk\" (UniqueName: \"kubernetes.io/projected/66eff972-2f71-45a8-af06-b5db71a265d0-kube-api-access-q2bpk\") pod \"66eff972-2f71-45a8-af06-b5db71a265d0\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") "
Nov 02 14:24:13.160985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.160218    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-metrics-certs\") pod \"66eff972-2f71-45a8-af06-b5db71a265d0\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") "
Nov 02 14:24:13.160985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.160297    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-maas-api-tls\") pod \"66eff972-2f71-45a8-af06-b5db71a265d0\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") "
Nov 02 14:24:13.160985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.160367    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66eff972-2f71-45a8-af06-b5db71a265d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66eff972-2f71-45a8-af06-b5db71a265d0-openshift-service-ca\") pod \"66eff972-2f71-45a8-af06-b5db71a265d0\" (UID: \"66eff972-2f71-45a8-af06-b5db71a265d0\") "
Nov 02 14:24:13.161572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.161539    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66eff972-2f71-45a8-af06-b5db71a265d0-openshift-service-ca" pod "66eff972-2f71-45a8-af06-b5db71a265d0" (UID: "66eff972-2f71-45a8-af06-b5db71a265d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:24:13.172516 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.172478    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-maas-api-tls" pod "66eff972-2f71-45a8-af06-b5db71a265d0" (UID: "66eff972-2f71-45a8-af06-b5db71a265d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:24:13.177131 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.176140    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-metrics-certs" pod "66eff972-2f71-45a8-af06-b5db71a265d0" (UID: "66eff972-2f71-45a8-af06-b5db71a265d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:24:13.184002 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.183958    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66eff972-2f71-45a8-af06-b5db71a265d0-kube-api-access-q2bpk" pod "66eff972-2f71-45a8-af06-b5db71a265d0" (UID: "66eff972-2f71-45a8-af06-b5db71a265d0"). InnerVolumeSpecName "kube-api-access-q2bpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:24:13.262445 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.262404    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:24:13.262445 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.262444    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66eff972-2f71-45a8-af06-b5db71a265d0-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:24:13.262720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.262461    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2bpk\" (UniqueName: \"kubernetes.io/projected/66eff972-2f71-45a8-af06-b5db71a265d0-kube-api-access-q2bpk\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:24:13.262720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.262474    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66eff972-2f71-45a8-af06-b5db71a265d0-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:24:13.612390 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.612355    6284 generic.go:356] "Generic (PLEG): container finished" podID="66eff972-2f71-45a8-af06-b5db71a265d0" containerID="f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87" exitCode=0
Nov 02 14:24:13.612609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.612414    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"
Nov 02 14:24:13.612609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.612421    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh" event={"ID":"66eff972-2f71-45a8-af06-b5db71a265d0","Type":"ContainerDied","Data":"f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87"}
Nov 02 14:24:13.612609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.612485    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh" event={"ID":"66eff972-2f71-45a8-af06-b5db71a265d0","Type":"ContainerDied","Data":"85701f5b7adb90e6460fd992d3841059f10cc36549b2433dcdaa2977f2f09fb1"}
Nov 02 14:24:13.612609 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.612513    6284 scope.go:122] "RemoveContainer" containerID="f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87"
Nov 02 14:24:13.623625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.623605    6284 scope.go:122] "RemoveContainer" containerID="f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87"
Nov 02 14:24:13.624007 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:24:13.623980    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87\": container with ID starting with f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87 not found: ID does not exist" containerID="f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87"
Nov 02 14:24:13.624065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.624018    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87"} err="failed to get container status \"f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87\": rpc error: code = NotFound desc = could not find container \"f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87\": container with ID starting with f6601b9ab1db2fbf5fc4f588a438848971e8d1afe687deb22235dcfe17571a87 not found: ID does not exist"
Nov 02 14:24:13.660386 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.660047    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"]
Nov 02 14:24:13.670855 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:13.670813    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0c2554-5dd87fd5d7-x4jsh"]
Nov 02 14:24:13.895304 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:24:13.895219    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:24:14.170336 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:14.170246    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66eff972-2f71-45a8-af06-b5db71a265d0" path="/var/lib/kubelet/pods/66eff972-2f71-45a8-af06-b5db71a265d0/volumes"
Nov 02 14:24:18.387347 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.387301    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"]
Nov 02 14:24:18.388188 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388167    6284 cpu_manager.go:414] "containerMap: removing container" podUID="8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" containerName="main"
Nov 02 14:24:18.388307 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388189    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" containerName="main"
Nov 02 14:24:18.388307 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388250    6284 cpu_manager.go:414] "containerMap: removing container" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy"
Nov 02 14:24:18.388307 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388259    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy"
Nov 02 14:24:18.388307 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388278    6284 cpu_manager.go:414] "containerMap: removing container" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerName="istio-proxy"
Nov 02 14:24:18.388307 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388286    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerName="istio-proxy"
Nov 02 14:24:18.388307 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388299    6284 cpu_manager.go:414] "containerMap: removing container" podUID="66eff972-2f71-45a8-af06-b5db71a265d0" containerName="maas-api"
Nov 02 14:24:18.388307 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388306    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66eff972-2f71-45a8-af06-b5db71a265d0" containerName="maas-api"
Nov 02 14:24:18.388644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388461    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d196b8-eeb0-4933-b320-2682b74cb13c" containerName="istio-proxy"
Nov 02 14:24:18.388644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388476    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24249a3-c205-40cf-b9d4-87dc53b97ff8" containerName="istio-proxy"
Nov 02 14:24:18.388644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388489    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="66eff972-2f71-45a8-af06-b5db71a265d0" containerName="maas-api"
Nov 02 14:24:18.388644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.388503    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="8caa0fb6-d69d-4bc9-ae0e-f40ee197aab9" containerName="main"
Nov 02 14:24:18.393326 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.393300    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.412397 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.412366    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"]
Nov 02 14:24:18.419295 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.419249    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-605541-serving-cert\""
Nov 02 14:24:18.531119 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.531078    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-maas-api-tls\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.531351 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.531259    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmqs\" (UniqueName: \"kubernetes.io/projected/897df9cf-6a88-42b3-9f22-61dcf942c079-kube-api-access-cpmqs\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.531428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.531396    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897df9cf-6a88-42b3-9f22-61dcf942c079-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.531470 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.531439    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-metrics-certs\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.632981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.632918    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-maas-api-tls\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.633223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.633078    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmqs\" (UniqueName: \"kubernetes.io/projected/897df9cf-6a88-42b3-9f22-61dcf942c079-kube-api-access-cpmqs\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.633223 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:24:18.633101    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-serving-cert: secret "maas-api-e2e-extmod-b-605541-serving-cert" not found
Nov 02 14:24:18.633223 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:24:18.633192    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-maas-api-tls podName:897df9cf-6a88-42b3-9f22-61dcf942c079 nodeName:}" failed. No retries permitted until 2026-11-02 14:24:19.133164564 +0000 UTC m=+2391.092540163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-maas-api-tls") pod "maas-api-e2e-extmod-b-605541-85686d6868-kb2r8" (UID: "897df9cf-6a88-42b3-9f22-61dcf942c079") : secret "maas-api-e2e-extmod-b-605541-serving-cert" not found
Nov 02 14:24:18.633412 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.633233    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897df9cf-6a88-42b3-9f22-61dcf942c079-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.633412 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.633295    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-metrics-certs\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.633920 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.633898    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897df9cf-6a88-42b3-9f22-61dcf942c079-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.636053 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.636027    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-metrics-certs\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:18.655105 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:18.655029    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmqs\" (UniqueName: \"kubernetes.io/projected/897df9cf-6a88-42b3-9f22-61dcf942c079-kube-api-access-cpmqs\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:19.139182 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:19.139149    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-maas-api-tls\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:19.141981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:19.141918    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-maas-api-tls\") pod \"maas-api-e2e-extmod-b-605541-85686d6868-kb2r8\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:19.312048 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:19.309451    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:19.650280 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:19.650171    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"]
Nov 02 14:24:20.666722 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:20.666681    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8" event={"ID":"897df9cf-6a88-42b3-9f22-61dcf942c079","Type":"ContainerStarted","Data":"61a7b961655295e805fc0bbdebd345375de6136a7614adc6b3ee458125b9c868"}
Nov 02 14:24:20.667209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:20.666747    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8" event={"ID":"897df9cf-6a88-42b3-9f22-61dcf942c079","Type":"ContainerStarted","Data":"e5fbcbc82207c327002d680c0bf6d3cfe7a59a9e38abe183427815249c874395"}
Nov 02 14:24:20.667209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:20.666852    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:20.703954 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:20.703872    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8" podStartSLOduration=2.276749817 podStartE2EDuration="2.70385365s" totalImagesPullingTime="427.103833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:20.703498802 +0000 UTC m=+2392.662874410" watchObservedRunningTime="2026-11-02 14:24:20.70385365 +0000 UTC m=+2392.663229261"
Nov 02 14:24:26.684596 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:26.684561    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:24:28.826379 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:28.826347    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:24:28.827340 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:24:28.827320    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:24:43.896486 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:24:43.896447    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:25:07.572979 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.572919    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"]
Nov 02 14:25:07.577308 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.577281    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.599416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.599383    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ce0f55-serving-cert\""
Nov 02 14:25:07.606020 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.605977    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"]
Nov 02 14:25:07.623897 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.623854    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.624126 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.624032    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92850818-3df8-445f-acda-7c17bf8b0577-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.624248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.624167    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5vn\" (UniqueName: \"kubernetes.io/projected/92850818-3df8-445f-acda-7c17bf8b0577-kube-api-access-km5vn\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.624248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.624205    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-metrics-certs\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.725757 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.725657    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92850818-3df8-445f-acda-7c17bf8b0577-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.725996 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.725964    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5vn\" (UniqueName: \"kubernetes.io/projected/92850818-3df8-445f-acda-7c17bf8b0577-kube-api-access-km5vn\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.726086 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.726034    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-metrics-certs\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.726086 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.726078    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.726284 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:25:07.726260    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-serving-cert: secret "maas-api-e2e-extmod-a-ce0f55-serving-cert" not found
Nov 02 14:25:07.726374 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:25:07.726362    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-maas-api-tls podName:92850818-3df8-445f-acda-7c17bf8b0577 nodeName:}" failed. No retries permitted until 2026-11-02 14:25:08.226336497 +0000 UTC m=+2440.185712101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-maas-api-tls") pod "maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj" (UID: "92850818-3df8-445f-acda-7c17bf8b0577") : secret "maas-api-e2e-extmod-a-ce0f55-serving-cert" not found
Nov 02 14:25:07.726448 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.726427    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92850818-3df8-445f-acda-7c17bf8b0577-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.728678 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.728647    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-metrics-certs\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:07.749779 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:07.749748    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5vn\" (UniqueName: \"kubernetes.io/projected/92850818-3df8-445f-acda-7c17bf8b0577-kube-api-access-km5vn\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:08.231616 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:08.231573    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:08.234040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:08.234005    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:08.493166 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:08.493118    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:08.808081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:08.808048    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"]
Nov 02 14:25:08.810708 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:25:08.810679    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92850818_3df8_445f_acda_7c17bf8b0577.slice/crio-935a276adc57cf9c13daee729c8fc09acc0a7654a3fb44ad43be81b37489d08b WatchSource:0}: Error finding container 935a276adc57cf9c13daee729c8fc09acc0a7654a3fb44ad43be81b37489d08b: Status 404 returned error can't find the container with id 935a276adc57cf9c13daee729c8fc09acc0a7654a3fb44ad43be81b37489d08b
Nov 02 14:25:09.006027 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:09.005981    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj" event={"ID":"92850818-3df8-445f-acda-7c17bf8b0577","Type":"ContainerStarted","Data":"935a276adc57cf9c13daee729c8fc09acc0a7654a3fb44ad43be81b37489d08b"}
Nov 02 14:25:10.014093 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:10.014049    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj" event={"ID":"92850818-3df8-445f-acda-7c17bf8b0577","Type":"ContainerStarted","Data":"65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3"}
Nov 02 14:25:10.014665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:10.014117    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:10.053243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:10.053182    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj" podStartSLOduration=2.7459481180000003 podStartE2EDuration="3.0531634s" totalImagesPullingTime="307.215282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:10.05103566 +0000 UTC m=+2442.010411268" watchObservedRunningTime="2026-11-02 14:25:10.0531634 +0000 UTC m=+2442.012539010"
Nov 02 14:25:13.895724 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:25:13.895680    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:25:16.027436 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:16.027402    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:25:43.896086 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:25:43.896043    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:25:54.248407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:54.248367    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"]
Nov 02 14:25:54.248906 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:54.248581    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy" containerID="cri-o://ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b" gracePeriod=30
Nov 02 14:25:57.909998 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:57.909920    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:25:57.910447 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:57.910022    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:25:59.597431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.597384    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 14:25:59.738190 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738157    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-envoy\") pod \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") "
Nov 02 14:25:59.738190 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738209    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-data\") pod \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") "
Nov 02 14:25:59.738473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738254    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-certs\") pod \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") "
Nov 02 14:25:59.738473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738344    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-credential-socket\") pod \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") "
Nov 02 14:25:59.738473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738395    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-token\" (UniqueName: \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-token\") pod \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") "
Nov 02 14:25:59.738473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738437    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-podinfo\") pod \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") "
Nov 02 14:25:59.738692 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738491    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istiod-ca-cert\") pod \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") "
Nov 02 14:25:59.738692 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738539    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-kube-api-access-85w8j\" (UniqueName: \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-kube-api-access-85w8j\") pod \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") "
Nov 02 14:25:59.738692 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738584    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-data" pod "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" (UID: "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:59.738692 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738605    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-certs" pod "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" (UID: "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:59.738692 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738683    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-socket\") pod \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\" (UID: \"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f\") "
Nov 02 14:25:59.738968 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.738848    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-credential-socket" pod "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" (UID: "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:59.739847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.739311    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:25:59.739847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.739339    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:25:59.739847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.739354    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:25:59.739847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.739530    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-socket" pod "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" (UID: "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:59.739847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.739781    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istiod-ca-cert" pod "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" (UID: "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:25:59.741277 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.741249    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-envoy" pod "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" (UID: "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:59.741414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.741316    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-token" pod "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" (UID: "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:59.741414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.741331    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-kube-api-access-85w8j" pod "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" (UID: "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f"). InnerVolumeSpecName "kube-api-access-85w8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:59.741645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.741620    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-podinfo" pod "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" (UID: "5985c62f-afea-4cd9-b2c1-7ebc41f24c5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:25:59.841644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.841531    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:25:59.841644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.841571    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:25:59.841644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.841585    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:25:59.841644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.841598    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:25:59.841644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.841611    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85w8j\" (UniqueName: \"kubernetes.io/projected/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-kube-api-access-85w8j\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:25:59.841644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:25:59.841627    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:26:00.372603 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:00.372556    6284 generic.go:356] "Generic (PLEG): container finished" podID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerID="ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b" exitCode=0
Nov 02 14:26:00.372789 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:00.372615    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"
Nov 02 14:26:00.372789 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:00.372631    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" event={"ID":"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f","Type":"ContainerDied","Data":"ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b"}
Nov 02 14:26:00.372789 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:00.372692    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f" event={"ID":"5985c62f-afea-4cd9-b2c1-7ebc41f24c5f","Type":"ContainerDied","Data":"9340062ea8fd726ce973ecd6ef059a752c480f4231cf84499bf71f6a9db0c1d5"}
Nov 02 14:26:00.372789 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:00.372719    6284 scope.go:122] "RemoveContainer" containerID="ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b"
Nov 02 14:26:00.383888 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:00.383708    6284 scope.go:122] "RemoveContainer" containerID="ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b"
Nov 02 14:26:00.384159 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:26:00.384136    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b\": container with ID starting with ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b not found: ID does not exist" containerID="ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b"
Nov 02 14:26:00.384235 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:00.384170    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b"} err="failed to get container status \"ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b\": rpc error: code = NotFound desc = could not find container \"ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b\": container with ID starting with ea06015d2b20dd71f135cfa042e19b29e96e59595e2e8985022a12133c566d0b not found: ID does not exist"
Nov 02 14:26:00.430042 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:00.430000    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"]
Nov 02 14:26:00.441058 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:00.440962    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-5d58f"]
Nov 02 14:26:02.171780 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:02.171749    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" path="/var/lib/kubelet/pods/5985c62f-afea-4cd9-b2c1-7ebc41f24c5f/volumes"
Nov 02 14:26:04.938116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:04.938075    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"]
Nov 02 14:26:04.938722 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:04.938707    6284 cpu_manager.go:414] "containerMap: removing container" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy"
Nov 02 14:26:04.938780 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:04.938722    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy"
Nov 02 14:26:04.938874 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:04.938864    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="5985c62f-afea-4cd9-b2c1-7ebc41f24c5f" containerName="istio-proxy"
Nov 02 14:26:04.944027 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:04.943997    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:26:04.966795 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:04.966761    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-66bc6d\""
Nov 02 14:26:04.968907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:04.968870    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"]
Nov 02 14:26:04.991953 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:04.991070    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-66bc6d-dockercfg-xkbbb\""
Nov 02 14:26:05.043099 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:05.043049    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8k2g\" (UniqueName: \"kubernetes.io/projected/91688d53-63bb-4a17-91c8-8e9d5f722eb4-kube-api-access-t8k2g\") pod \"payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t\" (UID: \"91688d53-63bb-4a17-91c8-8e9d5f722eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:26:05.043290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:05.043209    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91688d53-63bb-4a17-91c8-8e9d5f722eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t\" (UID: \"91688d53-63bb-4a17-91c8-8e9d5f722eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:26:05.144425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:05.144365    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8k2g\" (UniqueName: \"kubernetes.io/projected/91688d53-63bb-4a17-91c8-8e9d5f722eb4-kube-api-access-t8k2g\") pod \"payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t\" (UID: \"91688d53-63bb-4a17-91c8-8e9d5f722eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:26:05.144655 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:05.144493    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91688d53-63bb-4a17-91c8-8e9d5f722eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t\" (UID: \"91688d53-63bb-4a17-91c8-8e9d5f722eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:26:05.145702 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:05.145670    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91688d53-63bb-4a17-91c8-8e9d5f722eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t\" (UID: \"91688d53-63bb-4a17-91c8-8e9d5f722eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:26:05.165809 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:05.165769    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8k2g\" (UniqueName: \"kubernetes.io/projected/91688d53-63bb-4a17-91c8-8e9d5f722eb4-kube-api-access-t8k2g\") pod \"payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t\" (UID: \"91688d53-63bb-4a17-91c8-8e9d5f722eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:26:05.265753 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:05.265670    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:26:05.581513 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:05.581430    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"]
Nov 02 14:26:05.584451 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:26:05.584414    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91688d53_63bb_4a17_91c8_8e9d5f722eb4.slice/crio-ddeeb15b03b31ffe3f577dcfc4593bce553619727e306e189a0b91a4b1fbd315 WatchSource:0}: Error finding container ddeeb15b03b31ffe3f577dcfc4593bce553619727e306e189a0b91a4b1fbd315: Status 404 returned error can't find the container with id ddeeb15b03b31ffe3f577dcfc4593bce553619727e306e189a0b91a4b1fbd315
Nov 02 14:26:06.463453 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:06.463416    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t" event={"ID":"91688d53-63bb-4a17-91c8-8e9d5f722eb4","Type":"ContainerStarted","Data":"49c16cc5b16dca102c1883a34b6afb34ec5c82ee8e46f01bb70ebdad8f83b478"}
Nov 02 14:26:06.464005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:06.463479    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t" event={"ID":"91688d53-63bb-4a17-91c8-8e9d5f722eb4","Type":"ContainerStarted","Data":"ddeeb15b03b31ffe3f577dcfc4593bce553619727e306e189a0b91a4b1fbd315"}
Nov 02 14:26:06.464005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:06.463616    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:26:06.513067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:06.512998    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t" podStartSLOduration=2.512979824 podStartE2EDuration="2.512979824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:06.50918126 +0000 UTC m=+2498.468556866" watchObservedRunningTime="2026-11-02 14:26:06.512979824 +0000 UTC m=+2498.472355432"
Nov 02 14:26:08.035885 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.035846    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"]
Nov 02 14:26:08.047269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.047237    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.071562 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.071527    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"]
Nov 02 14:26:08.085374 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.085323    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.085581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.085430    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.085581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.085476    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.085581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.085520    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.085820 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.085592    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.085820 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.085617    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6sqt\" (UniqueName: \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-kube-api-access-h6sqt\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.085820 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.085669    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.085820 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.085703    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.086102 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.085822    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.141097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.141063    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ce0f55-openshift-default-dockercfg-ghzd9\""
Nov 02 14:26:08.187136 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187095    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.187136 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187142    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.187425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187279    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.187425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187385    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.187425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187415    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6sqt\" (UniqueName: \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-kube-api-access-h6sqt\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.187585 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187510    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.187585 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187513    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.187684 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187600    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.187770 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187737    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.187837 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187746    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.187898 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.187865    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.188084 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.188038    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.188209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.188181    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.188271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.188238    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.189855 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.189833    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.190441 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.190418    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.210974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.210900    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.211243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.211216    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6sqt\" (UniqueName: \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-kube-api-access-h6sqt\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.381717 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.381621    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:08.725338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.725296    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"]
Nov 02 14:26:08.727794 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:26:08.727755    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fc299a_3d6d_4da3_94de_a974f6e3afc1.slice/crio-ea467b9cd23760a2de2b85f61a6d1b19d3e475c326ab5e0829c2a40f445bbffb WatchSource:0}: Error finding container ea467b9cd23760a2de2b85f61a6d1b19d3e475c326ab5e0829c2a40f445bbffb: Status 404 returned error can't find the container with id ea467b9cd23760a2de2b85f61a6d1b19d3e475c326ab5e0829c2a40f445bbffb
Nov 02 14:26:08.730911 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.730871    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:26:08.731050 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.730989    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:26:08.731050 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:08.731025    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:26:09.487741 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:09.487703    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" event={"ID":"b8fc299a-3d6d-4da3-94de-a974f6e3afc1","Type":"ContainerStarted","Data":"0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e"}
Nov 02 14:26:09.488206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:09.487774    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" event={"ID":"b8fc299a-3d6d-4da3-94de-a974f6e3afc1","Type":"ContainerStarted","Data":"ea467b9cd23760a2de2b85f61a6d1b19d3e475c326ab5e0829c2a40f445bbffb"}
Nov 02 14:26:09.544707 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:09.544642    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" podStartSLOduration=1.544623152 podStartE2EDuration="1.544623152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:09.54234451 +0000 UTC m=+2501.501720118" watchObservedRunningTime="2026-11-02 14:26:09.544623152 +0000 UTC m=+2501.503998760"
Nov 02 14:26:10.382464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:10.382411    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:10.384160 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:10.384128    6284 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 14:26:10.384289 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:10.384186    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" podUID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 14:26:11.382285 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:11.382248    6284 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 14:26:11.382734 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:11.382310    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" podUID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 14:26:12.382518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:12.382477    6284 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 14:26:12.383009 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:12.382545    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" podUID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 14:26:13.382745 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:13.382708    6284 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 14:26:13.383223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:13.382769    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" podUID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 14:26:13.904814 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:26:13.904752    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:26:14.382383 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:14.382341    6284 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 14:26:14.382577 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:14.382418    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" podUID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 14:26:15.437342 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:15.437305    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:15.438352 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:15.438319    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:15.439432 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:15.439378    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:26:17.474033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:17.473998    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:26:33.056236 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.056197    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"]
Nov 02 14:26:33.062077 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.062047    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.077683 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.077647    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f30173-openshift-default-dockercfg-tw6wb\""
Nov 02 14:26:33.105579 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.105529    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"]
Nov 02 14:26:33.170243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.170198    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.170243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.170245    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.170477 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.170346    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.170477 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.170382    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.170547 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.170518    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.170589 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.170549    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca18629f-b465-4a85-87e3-bfc27af3e06a-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.170589 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.170581    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.170727 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.170700    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwmr5\" (UniqueName: \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-kube-api-access-nwmr5\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.170807 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.170794    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.272408 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.272361    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.272408 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.272411    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.272718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.272457    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.272718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.272479    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.272718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.272593    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.272718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.272654    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca18629f-b465-4a85-87e3-bfc27af3e06a-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.272930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.272732    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.272930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.272802    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwmr5\" (UniqueName: \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-kube-api-access-nwmr5\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.273114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.272969    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.273114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.272989    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.273114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.273074    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.273392 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.273366    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.273518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.273500    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.273563 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.273516    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca18629f-b465-4a85-87e3-bfc27af3e06a-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.275067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.275038    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.275265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.275244    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.301843 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.301791    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.302051 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.301906    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwmr5\" (UniqueName: \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-kube-api-access-nwmr5\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.377876 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.377767    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:33.708799 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.708770    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"]
Nov 02 14:26:33.711240 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:26:33.711212    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca18629f_b465_4a85_87e3_bfc27af3e06a.slice/crio-e9e5636eab6c4794df963fdc8826a346e53fa7d3aee7e405272a477dcd16c468 WatchSource:0}: Error finding container e9e5636eab6c4794df963fdc8826a346e53fa7d3aee7e405272a477dcd16c468: Status 404 returned error can't find the container with id e9e5636eab6c4794df963fdc8826a346e53fa7d3aee7e405272a477dcd16c468
Nov 02 14:26:33.714154 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.714115    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:26:33.714282 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.714239    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:26:33.714353 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:33.714293    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:26:34.690055 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:34.690014    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" event={"ID":"ca18629f-b465-4a85-87e3-bfc27af3e06a","Type":"ContainerStarted","Data":"7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6"}
Nov 02 14:26:34.690557 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:34.690073    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" event={"ID":"ca18629f-b465-4a85-87e3-bfc27af3e06a","Type":"ContainerStarted","Data":"e9e5636eab6c4794df963fdc8826a346e53fa7d3aee7e405272a477dcd16c468"}
Nov 02 14:26:34.747487 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:34.747430    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" podStartSLOduration=1.7474137220000001 podStartE2EDuration="1.747413722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:34.745516539 +0000 UTC m=+2526.704892146" watchObservedRunningTime="2026-11-02 14:26:34.747413722 +0000 UTC m=+2526.706789329"
Nov 02 14:26:35.378965 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:35.378887    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:35.380686 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:35.380657    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 14:26:35.380838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:35.380712    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 14:26:36.379014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:36.378979    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 14:26:36.379462 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:36.379035    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 14:26:37.378404 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:37.378367    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 14:26:37.378629 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:37.378425    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 14:26:38.379495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:38.379453    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 14:26:38.380110 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:38.380057    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 14:26:39.378199 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:39.378163    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 14:26:39.378390 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:39.378226    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 14:26:40.379249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:40.379209    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 14:26:40.379842 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:40.379281    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 14:26:41.432735 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:41.432703    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:41.433225 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:41.433201    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:41.434330 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:41.434306    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:26:43.895554 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:26:43.895517    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:26:43.958948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:43.958906    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"]
Nov 02 14:26:43.963898 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:43.963869    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:43.978641 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:43.978613    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-66bc6d-serving-cert\""
Nov 02 14:26:43.993563 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:43.993525    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"]
Nov 02 14:26:44.055141 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.055078    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-maas-api-tls\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.055350 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.055241    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-metrics-certs\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.055350 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.055304    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtxlb\" (UniqueName: \"kubernetes.io/projected/fc70a0fe-2fe3-41be-b674-f96892b5ae58-kube-api-access-qtxlb\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.055350 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.055327    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc70a0fe-2fe3-41be-b674-f96892b5ae58-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.156817 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.156728    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-maas-api-tls\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.156817 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.156813    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-metrics-certs\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.157124 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.156851    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtxlb\" (UniqueName: \"kubernetes.io/projected/fc70a0fe-2fe3-41be-b674-f96892b5ae58-kube-api-access-qtxlb\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.157124 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.156873    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc70a0fe-2fe3-41be-b674-f96892b5ae58-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.157628 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.157605    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc70a0fe-2fe3-41be-b674-f96892b5ae58-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.159491 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.159461    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-metrics-certs\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.159631 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.159585    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-maas-api-tls\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.210039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.202527    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtxlb\" (UniqueName: \"kubernetes.io/projected/fc70a0fe-2fe3-41be-b674-f96892b5ae58-kube-api-access-qtxlb\") pod \"maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.279686 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.279654    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:44.598623 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.598590    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"]
Nov 02 14:26:44.601750 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:26:44.601721    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc70a0fe_2fe3_41be_b674_f96892b5ae58.slice/crio-e48e081cf01c0339a571e439959d180f36bf935a4c59f27d4369355228a98659 WatchSource:0}: Error finding container e48e081cf01c0339a571e439959d180f36bf935a4c59f27d4369355228a98659: Status 404 returned error can't find the container with id e48e081cf01c0339a571e439959d180f36bf935a4c59f27d4369355228a98659
Nov 02 14:26:44.603391 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.603371    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:26:44.797262 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:44.797225    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd" event={"ID":"fc70a0fe-2fe3-41be-b674-f96892b5ae58","Type":"ContainerStarted","Data":"e48e081cf01c0339a571e439959d180f36bf935a4c59f27d4369355228a98659"}
Nov 02 14:26:45.808277 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:45.808236    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd" event={"ID":"fc70a0fe-2fe3-41be-b674-f96892b5ae58","Type":"ContainerStarted","Data":"d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b"}
Nov 02 14:26:45.808758 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:45.808388    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:26:45.857375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:45.857286    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd" podStartSLOduration=2.4708942179999998 podStartE2EDuration="2.857260866s" totalImagesPullingTime="386.366648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:45.853155018 +0000 UTC m=+2537.812530626" watchObservedRunningTime="2026-11-02 14:26:45.857260866 +0000 UTC m=+2537.816636474"
Nov 02 14:26:51.821259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:26:51.821221    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:27:13.897915 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:27:13.897870    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:27:43.896548 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:27:43.896508    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:27:56.273711 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.273624    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"]
Nov 02 14:27:56.278512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.278481    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.308657 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.308618    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"agc-cred-c56b8ba7682d\""
Nov 02 14:27:56.308657 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.308620    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bc34db\""
Nov 02 14:27:56.308657 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.308618    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"openshift-service-ca.crt\""
Nov 02 14:27:56.309018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.308755    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"routing-overlay\""
Nov 02 14:27:56.309018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.308756    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"agc-cred-4c908c3e55be\""
Nov 02 14:27:56.309018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.308813    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"payload-processing-external-model-e2e--2bdbc850-dockercfg-v8bp5\""
Nov 02 14:27:56.309018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.308836    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"kube-root-ca.crt\""
Nov 02 14:27:56.314139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.314095    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"]
Nov 02 14:27:56.333873 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.333813    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.334140 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.334037    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svn7\" (UniqueName: \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-kube-api-access-8svn7\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.334140 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.334080    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.334140 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.334106    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.435529 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.435476    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.435745 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.435663    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svn7\" (UniqueName: \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-kube-api-access-8svn7\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.435745 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.435721    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.435870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.435749    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.436397 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.436366    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.436783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.436756    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.438862 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.438837    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.457274 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.457244    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svn7\" (UniqueName: \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-kube-api-access-8svn7\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.594641 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.594544    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:56.931587 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:56.931556    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"]
Nov 02 14:27:56.933891 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:27:56.933862    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbdf14b_ae14_4d41_b573_37e43f798aaf.slice/crio-377f029650257ba155665f74c88baa00b962e8973e32f0ce85dd727153c4b1c8 WatchSource:0}: Error finding container 377f029650257ba155665f74c88baa00b962e8973e32f0ce85dd727153c4b1c8: Status 404 returned error can't find the container with id 377f029650257ba155665f74c88baa00b962e8973e32f0ce85dd727153c4b1c8
Nov 02 14:27:57.355821 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:57.355771    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb" event={"ID":"8fbdf14b-ae14-4d41-b573-37e43f798aaf","Type":"ContainerStarted","Data":"45290c61c1622fbd03b07c1ed749a9cb10b50a760d446032e99202b3dae47b30"}
Nov 02 14:27:57.356309 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:57.355883    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:27:57.356309 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:57.355907    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb" event={"ID":"8fbdf14b-ae14-4d41-b573-37e43f798aaf","Type":"ContainerStarted","Data":"377f029650257ba155665f74c88baa00b962e8973e32f0ce85dd727153c4b1c8"}
Nov 02 14:27:57.394005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:27:57.393838    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb" podStartSLOduration=1.393818788 podStartE2EDuration="1.393818788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:57.392543727 +0000 UTC m=+2609.351919335" watchObservedRunningTime="2026-11-02 14:27:57.393818788 +0000 UTC m=+2609.353194395"
Nov 02 14:28:07.302308 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.302269    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"]
Nov 02 14:28:07.305734 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.305701    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:28:07.341167 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.341127    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"]
Nov 02 14:28:07.345814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.345778    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.365878 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.365843    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"]
Nov 02 14:28:07.492869 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.492826    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.493103 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.492946    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.493103 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.493025    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsbmb\" (UniqueName: \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-kube-api-access-vsbmb\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.493221 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.493106    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.597998 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.597832    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.597998 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.597964    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.598255 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.598043    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsbmb\" (UniqueName: \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-kube-api-access-vsbmb\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.598255 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.598123    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.598949 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.598902    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.600348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.600313    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.602027 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.601998    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.620904 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.620873    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsbmb\" (UniqueName: \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-kube-api-access-vsbmb\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.663546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.663504    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:07.976915 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:07.976881    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"]
Nov 02 14:28:07.979624 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:28:07.979591    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428e2c3e_fa01_4b9a_ace4_1738ef601501.slice/crio-e91c45e97b7d03c3f5d4773c00150fdcc02c45c50dc3500403f598ec280a92cd WatchSource:0}: Error finding container e91c45e97b7d03c3f5d4773c00150fdcc02c45c50dc3500403f598ec280a92cd: Status 404 returned error can't find the container with id e91c45e97b7d03c3f5d4773c00150fdcc02c45c50dc3500403f598ec280a92cd
Nov 02 14:28:08.435688 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:08.435650    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl" event={"ID":"428e2c3e-fa01-4b9a-ace4-1738ef601501","Type":"ContainerStarted","Data":"25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862"}
Nov 02 14:28:08.436137 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:08.435710    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl" event={"ID":"428e2c3e-fa01-4b9a-ace4-1738ef601501","Type":"ContainerStarted","Data":"e91c45e97b7d03c3f5d4773c00150fdcc02c45c50dc3500403f598ec280a92cd"}
Nov 02 14:28:08.436137 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:08.435734    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:08.473557 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:08.473469    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl" podStartSLOduration=1.473450891 podStartE2EDuration="1.473450891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:08.471924872 +0000 UTC m=+2620.431300494" watchObservedRunningTime="2026-11-02 14:28:08.473450891 +0000 UTC m=+2620.432826500"
Nov 02 14:28:12.303069 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.303010    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb" podUID="8fbdf14b-ae14-4d41-b573-37e43f798aaf" containerName="payload-processing" containerID="cri-o://45290c61c1622fbd03b07c1ed749a9cb10b50a760d446032e99202b3dae47b30" gracePeriod=25
Nov 02 14:28:12.466441 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.466409    6284 generic.go:356] "Generic (PLEG): container finished" podID="8fbdf14b-ae14-4d41-b573-37e43f798aaf" containerID="45290c61c1622fbd03b07c1ed749a9cb10b50a760d446032e99202b3dae47b30" exitCode=0
Nov 02 14:28:12.466614 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.466474    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb" event={"ID":"8fbdf14b-ae14-4d41-b573-37e43f798aaf","Type":"ContainerDied","Data":"45290c61c1622fbd03b07c1ed749a9cb10b50a760d446032e99202b3dae47b30"}
Nov 02 14:28:12.572924 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.572850    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:28:12.585624 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.585597    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-provider-credentials\") pod \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") "
Nov 02 14:28:12.585883 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.585863    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-kube-api-access-8svn7\" (UniqueName: \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-kube-api-access-8svn7\") pod \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") "
Nov 02 14:28:12.585979 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.585907    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-plugins-config-volume\") pod \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") "
Nov 02 14:28:12.586036 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.585984    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-routing-overlay-volume\") pod \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\" (UID: \"8fbdf14b-ae14-4d41-b573-37e43f798aaf\") "
Nov 02 14:28:12.586404 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.586379    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-routing-overlay-volume" pod "8fbdf14b-ae14-4d41-b573-37e43f798aaf" (UID: "8fbdf14b-ae14-4d41-b573-37e43f798aaf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:12.586494 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.586398    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-plugins-config-volume" pod "8fbdf14b-ae14-4d41-b573-37e43f798aaf" (UID: "8fbdf14b-ae14-4d41-b573-37e43f798aaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:12.586560 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.586542    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:28:12.586600 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.586569    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbdf14b-ae14-4d41-b573-37e43f798aaf-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:28:12.588109 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.588089    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-provider-credentials" pod "8fbdf14b-ae14-4d41-b573-37e43f798aaf" (UID: "8fbdf14b-ae14-4d41-b573-37e43f798aaf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:12.588178 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.588159    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-kube-api-access-8svn7" pod "8fbdf14b-ae14-4d41-b573-37e43f798aaf" (UID: "8fbdf14b-ae14-4d41-b573-37e43f798aaf"). InnerVolumeSpecName "kube-api-access-8svn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:12.687714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.687677    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8svn7\" (UniqueName: \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-kube-api-access-8svn7\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:28:12.687714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:12.687710    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbdf14b-ae14-4d41-b573-37e43f798aaf-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:28:13.478266 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:13.478224    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"
Nov 02 14:28:13.478266 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:13.478253    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb" event={"ID":"8fbdf14b-ae14-4d41-b573-37e43f798aaf","Type":"ContainerDied","Data":"377f029650257ba155665f74c88baa00b962e8973e32f0ce85dd727153c4b1c8"}
Nov 02 14:28:13.478870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:13.478333    6284 scope.go:122] "RemoveContainer" containerID="45290c61c1622fbd03b07c1ed749a9cb10b50a760d446032e99202b3dae47b30"
Nov 02 14:28:13.524685 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:13.524645    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"]
Nov 02 14:28:13.538697 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:13.538645    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5fzjvb"]
Nov 02 14:28:13.897155 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:28:13.897028    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:28:14.170435 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:14.170351    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbdf14b-ae14-4d41-b573-37e43f798aaf" path="/var/lib/kubelet/pods/8fbdf14b-ae14-4d41-b573-37e43f798aaf/volumes"
Nov 02 14:28:18.982688 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:18.982620    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"]
Nov 02 14:28:18.987717 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:18.987633    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:23.983142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:23.983062    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl" podUID="428e2c3e-fa01-4b9a-ace4-1738ef601501" containerName="payload-processing" containerID="cri-o://25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862" gracePeriod=25
Nov 02 14:28:24.257896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.257864    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:24.351737 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.351695    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-routing-overlay-volume\") pod \"428e2c3e-fa01-4b9a-ace4-1738ef601501\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") "
Nov 02 14:28:24.351972 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.351807    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-plugins-config-volume\") pod \"428e2c3e-fa01-4b9a-ace4-1738ef601501\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") "
Nov 02 14:28:24.351972 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.351872    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-provider-credentials\" (UniqueName: \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-provider-credentials\") pod \"428e2c3e-fa01-4b9a-ace4-1738ef601501\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") "
Nov 02 14:28:24.352152 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.351982    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-kube-api-access-vsbmb\" (UniqueName: \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-kube-api-access-vsbmb\") pod \"428e2c3e-fa01-4b9a-ace4-1738ef601501\" (UID: \"428e2c3e-fa01-4b9a-ace4-1738ef601501\") "
Nov 02 14:28:24.352292 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.352267    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-routing-overlay-volume" pod "428e2c3e-fa01-4b9a-ace4-1738ef601501" (UID: "428e2c3e-fa01-4b9a-ace4-1738ef601501"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:24.352382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.352296    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-plugins-config-volume" pod "428e2c3e-fa01-4b9a-ace4-1738ef601501" (UID: "428e2c3e-fa01-4b9a-ace4-1738ef601501"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:24.352717 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.352693    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:28:24.352816 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.352724    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428e2c3e-fa01-4b9a-ace4-1738ef601501-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:28:24.354355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.354326    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-provider-credentials" pod "428e2c3e-fa01-4b9a-ace4-1738ef601501" (UID: "428e2c3e-fa01-4b9a-ace4-1738ef601501"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:24.354454 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.354372    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-kube-api-access-vsbmb" pod "428e2c3e-fa01-4b9a-ace4-1738ef601501" (UID: "428e2c3e-fa01-4b9a-ace4-1738ef601501"). InnerVolumeSpecName "kube-api-access-vsbmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:24.453701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.453651    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:28:24.453701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.453695    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsbmb\" (UniqueName: \"kubernetes.io/projected/428e2c3e-fa01-4b9a-ace4-1738ef601501-kube-api-access-vsbmb\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:28:24.562378 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.562281    6284 generic.go:356] "Generic (PLEG): container finished" podID="428e2c3e-fa01-4b9a-ace4-1738ef601501" containerID="25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862" exitCode=0
Nov 02 14:28:24.562378 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.562344    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"
Nov 02 14:28:24.562378 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.562363    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl" event={"ID":"428e2c3e-fa01-4b9a-ace4-1738ef601501","Type":"ContainerDied","Data":"25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862"}
Nov 02 14:28:24.562637 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.562427    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl" event={"ID":"428e2c3e-fa01-4b9a-ace4-1738ef601501","Type":"ContainerDied","Data":"e91c45e97b7d03c3f5d4773c00150fdcc02c45c50dc3500403f598ec280a92cd"}
Nov 02 14:28:24.562637 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.562455    6284 scope.go:122] "RemoveContainer" containerID="25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862"
Nov 02 14:28:24.574851 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.574829    6284 scope.go:122] "RemoveContainer" containerID="25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862"
Nov 02 14:28:24.575236 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:28:24.575212    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862\": container with ID starting with 25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862 not found: ID does not exist" containerID="25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862"
Nov 02 14:28:24.575323 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.575246    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862"} err="failed to get container status \"25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862\": rpc error: code = NotFound desc = could not find container \"25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862\": container with ID starting with 25d9475ed50deb96e4c9dd76e31c72699d9a28531e987be34d8dffcd2ab98862 not found: ID does not exist"
Nov 02 14:28:24.607993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.607922    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"]
Nov 02 14:28:24.614697 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:24.614662    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987545wl"]
Nov 02 14:28:26.183306 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:26.183266    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428e2c3e-fa01-4b9a-ace4-1738ef601501" path="/var/lib/kubelet/pods/428e2c3e-fa01-4b9a-ace4-1738ef601501/volumes"
Nov 02 14:28:43.896610 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:28:43.896564    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:28:56.577952 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.577842    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"]
Nov 02 14:28:56.578698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.578669    6284 cpu_manager.go:414] "containerMap: removing container" podUID="8fbdf14b-ae14-4d41-b573-37e43f798aaf" containerName="payload-processing"
Nov 02 14:28:56.578698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.578698    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbdf14b-ae14-4d41-b573-37e43f798aaf" containerName="payload-processing"
Nov 02 14:28:56.578971 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.578730    6284 cpu_manager.go:414] "containerMap: removing container" podUID="428e2c3e-fa01-4b9a-ace4-1738ef601501" containerName="payload-processing"
Nov 02 14:28:56.578971 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.578738    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428e2c3e-fa01-4b9a-ace4-1738ef601501" containerName="payload-processing"
Nov 02 14:28:56.578971 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.578958    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbdf14b-ae14-4d41-b573-37e43f798aaf" containerName="payload-processing"
Nov 02 14:28:56.579146 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.578979    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="428e2c3e-fa01-4b9a-ace4-1738ef601501" containerName="payload-processing"
Nov 02 14:28:56.583386 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.583361    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.607926 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.607895    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"agc-cred-9b11dc31ee6b\""
Nov 02 14:28:56.608406 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.607946    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"agc-cred-cc84e5143c90\""
Nov 02 14:28:56.608548 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.608165    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"openshift-service-ca.crt\""
Nov 02 14:28:56.608548 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.608241    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"kube-root-ca.crt\""
Nov 02 14:28:56.608548 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.608275    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-66bc6d\""
Nov 02 14:28:56.608548 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.608492    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"routing-overlay\""
Nov 02 14:28:56.608773 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.608562    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"payload-processing-external-model-e2e--47c96fd3-dockercfg-6h7wl\""
Nov 02 14:28:56.613134 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.613098    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"]
Nov 02 14:28:56.770579 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.770531    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.770769 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.770588    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.770814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.770773    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.770859 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.770848    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grtql\" (UniqueName: \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-kube-api-access-grtql\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.872429 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.872340    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grtql\" (UniqueName: \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-kube-api-access-grtql\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.872604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.872481    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.872604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.872519    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.872693 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.872662    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.873339 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.873317    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.873443 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.873422    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.875178 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.875155    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.893913 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.893867    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grtql\" (UniqueName: \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-kube-api-access-grtql\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:56.898706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:56.898670    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:57.210695 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.210594    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"]
Nov 02 14:28:57.213840 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:28:57.213800    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f529582_7483_43c5_b333_cbc48832a755.slice/crio-e7ed29505d30c374031ce98291f41335f4aa21c36cbf7248406676769e8faa9a WatchSource:0}: Error finding container e7ed29505d30c374031ce98291f41335f4aa21c36cbf7248406676769e8faa9a: Status 404 returned error can't find the container with id e7ed29505d30c374031ce98291f41335f4aa21c36cbf7248406676769e8faa9a
Nov 02 14:28:57.744240 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.744201    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"]
Nov 02 14:28:57.749041 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.749015    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.769772 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.769729    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"agc-cred-a5feac353083\""
Nov 02 14:28:57.770090 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.769772    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"payload-processing-external-model-e2e--aa97af6d-dockercfg-2jkwn\""
Nov 02 14:28:57.770488 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.770461    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"openshift-service-ca.crt\""
Nov 02 14:28:57.770488 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.770462    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ce0f55\""
Nov 02 14:28:57.770665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.770528    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"kube-root-ca.crt\""
Nov 02 14:28:57.770665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.770589    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ce0f55\"/\"routing-overlay\""
Nov 02 14:28:57.774198 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.774166    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"]
Nov 02 14:28:57.787229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.787189    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.787414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.787385    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.787462 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.787422    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcpnr\" (UniqueName: \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-kube-api-access-gcpnr\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.787665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.787636    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.837186 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.837143    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9" event={"ID":"6f529582-7483-43c5-b333-cbc48832a755","Type":"ContainerStarted","Data":"ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9"}
Nov 02 14:28:57.837407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.837216    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9" event={"ID":"6f529582-7483-43c5-b333-cbc48832a755","Type":"ContainerStarted","Data":"e7ed29505d30c374031ce98291f41335f4aa21c36cbf7248406676769e8faa9a"}
Nov 02 14:28:57.837407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.837309    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:28:57.901006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.897279    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.901006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.897471    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.901006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.897514    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcpnr\" (UniqueName: \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-kube-api-access-gcpnr\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.901006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.897646    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.904337 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.902396    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.905616 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.905422    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.908668 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.908614    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:57.933921 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:57.933880    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcpnr\" (UniqueName: \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-kube-api-access-gcpnr\") pod \"payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") " pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:58.063978 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:58.063835    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:58.392993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:58.392867    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9" podStartSLOduration=2.392849669 podStartE2EDuration="2.392849669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:57.882102279 +0000 UTC m=+2669.841477887" watchObservedRunningTime="2026-11-02 14:28:58.392849669 +0000 UTC m=+2670.352225275"
Nov 02 14:28:58.394497 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:58.394425    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"]
Nov 02 14:28:58.846256 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:58.846209    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd" event={"ID":"673977e8-ad5e-4bd6-bba4-14b1de66b0fc","Type":"ContainerStarted","Data":"f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f"}
Nov 02 14:28:58.846795 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:58.846285    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:28:58.846795 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:58.846314    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd" event={"ID":"673977e8-ad5e-4bd6-bba4-14b1de66b0fc","Type":"ContainerStarted","Data":"95cdd9369ad309a28dfecda8c93508e79d39a3b0b338f19c5c813bdbdb8c8ed8"}
Nov 02 14:28:58.884992 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:28:58.884906    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd" podStartSLOduration=1.884883076 podStartE2EDuration="1.884883076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:28:58.88310513 +0000 UTC m=+2670.842480729" watchObservedRunningTime="2026-11-02 14:28:58.884883076 +0000 UTC m=+2670.844258684"
Nov 02 14:29:02.408563 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:02.408511    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"]
Nov 02 14:29:03.882372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:03.882331    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"]
Nov 02 14:29:03.885894 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:03.885862    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:29:07.409816 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.409741    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd" podUID="673977e8-ad5e-4bd6-bba4-14b1de66b0fc" containerName="payload-processing" containerID="cri-o://f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f" gracePeriod=25
Nov 02 14:29:07.692376 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.692343    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:29:07.874366 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.874327    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-provider-credentials\") pod \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") "
Nov 02 14:29:07.874581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.874448    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-routing-overlay-volume\") pod \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") "
Nov 02 14:29:07.874581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.874507    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-kube-api-access-gcpnr\" (UniqueName: \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-kube-api-access-gcpnr\") pod \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") "
Nov 02 14:29:07.874581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.874576    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-plugins-config-volume\") pod \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\" (UID: \"673977e8-ad5e-4bd6-bba4-14b1de66b0fc\") "
Nov 02 14:29:07.874924 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.874896    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-routing-overlay-volume" pod "673977e8-ad5e-4bd6-bba4-14b1de66b0fc" (UID: "673977e8-ad5e-4bd6-bba4-14b1de66b0fc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:07.875093 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.875071    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-plugins-config-volume" pod "673977e8-ad5e-4bd6-bba4-14b1de66b0fc" (UID: "673977e8-ad5e-4bd6-bba4-14b1de66b0fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:07.875166 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.875128    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:07.876866 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.876835    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-provider-credentials" pod "673977e8-ad5e-4bd6-bba4-14b1de66b0fc" (UID: "673977e8-ad5e-4bd6-bba4-14b1de66b0fc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:07.877213 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.877193    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-kube-api-access-gcpnr" pod "673977e8-ad5e-4bd6-bba4-14b1de66b0fc" (UID: "673977e8-ad5e-4bd6-bba4-14b1de66b0fc"). InnerVolumeSpecName "kube-api-access-gcpnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:07.945605 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.945517    6284 generic.go:356] "Generic (PLEG): container finished" podID="673977e8-ad5e-4bd6-bba4-14b1de66b0fc" containerID="f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f" exitCode=0
Nov 02 14:29:07.945605 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.945589    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"
Nov 02 14:29:07.945838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.945603    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd" event={"ID":"673977e8-ad5e-4bd6-bba4-14b1de66b0fc","Type":"ContainerDied","Data":"f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f"}
Nov 02 14:29:07.945838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.945667    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd" event={"ID":"673977e8-ad5e-4bd6-bba4-14b1de66b0fc","Type":"ContainerDied","Data":"95cdd9369ad309a28dfecda8c93508e79d39a3b0b338f19c5c813bdbdb8c8ed8"}
Nov 02 14:29:07.945838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.945694    6284 scope.go:122] "RemoveContainer" containerID="f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f"
Nov 02 14:29:07.958983 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.958963    6284 scope.go:122] "RemoveContainer" containerID="f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f"
Nov 02 14:29:07.959400 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:29:07.959373    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f\": container with ID starting with f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f not found: ID does not exist" containerID="f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f"
Nov 02 14:29:07.959460 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.959409    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f"} err="failed to get container status \"f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f\": rpc error: code = NotFound desc = could not find container \"f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f\": container with ID starting with f6a9ded7f0346b6a9c095f97a95e98d86d145ca7c9f5637a6b46ef5ec927e98f not found: ID does not exist"
Nov 02 14:29:07.976207 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.976171    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:07.976207 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.976200    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:07.976207 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.976213    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcpnr\" (UniqueName: \"kubernetes.io/projected/673977e8-ad5e-4bd6-bba4-14b1de66b0fc-kube-api-access-gcpnr\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:07.997163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:07.997119    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"]
Nov 02 14:29:08.003847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:08.003811    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ce0f55/payload-processing-external-model-e2e-extmod-a-ce0f55-5c55ffbxd"]
Nov 02 14:29:08.172595 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:08.172548    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673977e8-ad5e-4bd6-bba4-14b1de66b0fc" path="/var/lib/kubelet/pods/673977e8-ad5e-4bd6-bba4-14b1de66b0fc/volumes"
Nov 02 14:29:08.882993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:08.882913    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9" podUID="6f529582-7483-43c5-b333-cbc48832a755" containerName="payload-processing" containerID="cri-o://ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9" gracePeriod=25
Nov 02 14:29:09.158040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.158009    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:29:09.192844 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.192798    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-plugins-config-volume\") pod \"6f529582-7483-43c5-b333-cbc48832a755\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") "
Nov 02 14:29:09.193103 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.192868    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-routing-overlay-volume\") pod \"6f529582-7483-43c5-b333-cbc48832a755\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") "
Nov 02 14:29:09.193103 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.193008    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-kube-api-access-grtql\" (UniqueName: \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-kube-api-access-grtql\") pod \"6f529582-7483-43c5-b333-cbc48832a755\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") "
Nov 02 14:29:09.193103 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.193073    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-provider-credentials\") pod \"6f529582-7483-43c5-b333-cbc48832a755\" (UID: \"6f529582-7483-43c5-b333-cbc48832a755\") "
Nov 02 14:29:09.193319 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.193287    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-routing-overlay-volume" pod "6f529582-7483-43c5-b333-cbc48832a755" (UID: "6f529582-7483-43c5-b333-cbc48832a755"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:09.193381 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.193291    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-plugins-config-volume" pod "6f529582-7483-43c5-b333-cbc48832a755" (UID: "6f529582-7483-43c5-b333-cbc48832a755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:09.193591 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.193573    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:09.193660 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.193596    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f529582-7483-43c5-b333-cbc48832a755-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:09.195696 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.195660    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-provider-credentials" pod "6f529582-7483-43c5-b333-cbc48832a755" (UID: "6f529582-7483-43c5-b333-cbc48832a755"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:09.195828 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.195721    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-kube-api-access-grtql" pod "6f529582-7483-43c5-b333-cbc48832a755" (UID: "6f529582-7483-43c5-b333-cbc48832a755"). InnerVolumeSpecName "kube-api-access-grtql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:09.294813 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.294773    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grtql\" (UniqueName: \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-kube-api-access-grtql\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:09.294813 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.294808    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f529582-7483-43c5-b333-cbc48832a755-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:09.962030 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.961988    6284 generic.go:356] "Generic (PLEG): container finished" podID="6f529582-7483-43c5-b333-cbc48832a755" containerID="ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9" exitCode=0
Nov 02 14:29:09.962532 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.962038    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9" event={"ID":"6f529582-7483-43c5-b333-cbc48832a755","Type":"ContainerDied","Data":"ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9"}
Nov 02 14:29:09.962532 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.962071    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"
Nov 02 14:29:09.962532 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.962145    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9" event={"ID":"6f529582-7483-43c5-b333-cbc48832a755","Type":"ContainerDied","Data":"e7ed29505d30c374031ce98291f41335f4aa21c36cbf7248406676769e8faa9a"}
Nov 02 14:29:09.962532 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.962180    6284 scope.go:122] "RemoveContainer" containerID="ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9"
Nov 02 14:29:09.973299 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.973274    6284 scope.go:122] "RemoveContainer" containerID="ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9"
Nov 02 14:29:09.973656 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:29:09.973628    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9\": container with ID starting with ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9 not found: ID does not exist" containerID="ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9"
Nov 02 14:29:09.973740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:09.973668    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9"} err="failed to get container status \"ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9\": rpc error: code = NotFound desc = could not find container \"ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9\": container with ID starting with ec38852c0da97300a50047307e51c9b73ab49ad91ee5e7a8a43893b5bf73a2c9 not found: ID does not exist"
Nov 02 14:29:10.007427 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:10.007389    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"]
Nov 02 14:29:10.017050 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:10.017009    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-57d8p79q9"]
Nov 02 14:29:10.171699 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:10.171663    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f529582-7483-43c5-b333-cbc48832a755" path="/var/lib/kubelet/pods/6f529582-7483-43c5-b333-cbc48832a755/volumes"
Nov 02 14:29:13.965868 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:29:13.965823    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:29:17.637274 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.637232    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"]
Nov 02 14:29:17.637926 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.637907    6284 cpu_manager.go:414] "containerMap: removing container" podUID="673977e8-ad5e-4bd6-bba4-14b1de66b0fc" containerName="payload-processing"
Nov 02 14:29:17.637926 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.637926    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673977e8-ad5e-4bd6-bba4-14b1de66b0fc" containerName="payload-processing"
Nov 02 14:29:17.638101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.637997    6284 cpu_manager.go:414] "containerMap: removing container" podUID="6f529582-7483-43c5-b333-cbc48832a755" containerName="payload-processing"
Nov 02 14:29:17.638101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.638006    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f529582-7483-43c5-b333-cbc48832a755" containerName="payload-processing"
Nov 02 14:29:17.638198 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.638184    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="673977e8-ad5e-4bd6-bba4-14b1de66b0fc" containerName="payload-processing"
Nov 02 14:29:17.638234 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.638206    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f529582-7483-43c5-b333-cbc48832a755" containerName="payload-processing"
Nov 02 14:29:17.643282 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.643254    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.667720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.667651    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"openshift-service-ca.crt\""
Nov 02 14:29:17.667720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.667668    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"agc-cred-1247e009eddc\""
Nov 02 14:29:17.667720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.667661    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"routing-overlay\""
Nov 02 14:29:17.667720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.667662    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-5f98a6\""
Nov 02 14:29:17.668204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.668182    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"payload-processing-external-model-e2e--300d9f08-dockercfg-fg5j4\""
Nov 02 14:29:17.668474 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.668455    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"kube-root-ca.crt\""
Nov 02 14:29:17.674062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.674030    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"]
Nov 02 14:29:17.725257 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.725222    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdwq6\" (UniqueName: \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-kube-api-access-rdwq6\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.725257 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.725269    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.725555 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.725289    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.725555 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.725494    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.826807 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.826766    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdwq6\" (UniqueName: \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-kube-api-access-rdwq6\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.826807 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.826814    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.827159 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.826928    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.827159 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.827085    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.827653 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.827632    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.827701 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.827649    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.829516 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.829484    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.884473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.883217    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdwq6\" (UniqueName: \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-kube-api-access-rdwq6\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:17.960305 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:17.960210    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:18.280018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.279989    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"]
Nov 02 14:29:18.283160 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:29:18.283124    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31810e37_907c_481e_be84_c9399536b616.slice/crio-02f959d8d61a3a86ce5f0066bb573f974f2b68a451879e147b53b7320c0b820f WatchSource:0}: Error finding container 02f959d8d61a3a86ce5f0066bb573f974f2b68a451879e147b53b7320c0b820f: Status 404 returned error can't find the container with id 02f959d8d61a3a86ce5f0066bb573f974f2b68a451879e147b53b7320c0b820f
Nov 02 14:29:18.397674 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.397628    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"]
Nov 02 14:29:18.402139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.402110    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.420363 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.420331    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-5f98a6\"/\"agc-cred-52e53de3b6d7\""
Nov 02 14:29:18.425013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.424974    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"]
Nov 02 14:29:18.438571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.438521    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.438760 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.438585    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.438760 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.438718    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.439069 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.439047    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4b85\" (UniqueName: \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-kube-api-access-q4b85\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.540105 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.539999    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4b85\" (UniqueName: \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-kube-api-access-q4b85\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.540285 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.540183    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.540285 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.540227    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.540285 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.540269    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.540905 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.540878    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.541055 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.541035    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.542846 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.542829    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.561267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.561234    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4b85\" (UniqueName: \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-kube-api-access-q4b85\") pod \"payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") " pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:18.717559 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:18.717515    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:19.036822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:19.036777    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m" event={"ID":"31810e37-907c-481e-be84-c9399536b616","Type":"ContainerStarted","Data":"0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523"}
Nov 02 14:29:19.037071 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:19.036881    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m" event={"ID":"31810e37-907c-481e-be84-c9399536b616","Type":"ContainerStarted","Data":"02f959d8d61a3a86ce5f0066bb573f974f2b68a451879e147b53b7320c0b820f"}
Nov 02 14:29:19.037071 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:19.037037    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:19.062416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:19.062385    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"]
Nov 02 14:29:19.064160 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:29:19.064128    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434a8e25_6856_46f2_b0ea_12d675c9420d.slice/crio-6f065a283cbc05f826e6e302d44b85fd7e3a5d5681e92b19f0a42bdb909bc3a6 WatchSource:0}: Error finding container 6f065a283cbc05f826e6e302d44b85fd7e3a5d5681e92b19f0a42bdb909bc3a6: Status 404 returned error can't find the container with id 6f065a283cbc05f826e6e302d44b85fd7e3a5d5681e92b19f0a42bdb909bc3a6
Nov 02 14:29:19.120486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:19.120432    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m" podStartSLOduration=2.120415631 podStartE2EDuration="2.120415631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:19.116801147 +0000 UTC m=+2691.076176756" watchObservedRunningTime="2026-11-02 14:29:19.120415631 +0000 UTC m=+2691.079791278"
Nov 02 14:29:20.045248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:20.045208    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz" event={"ID":"434a8e25-6856-46f2-b0ea-12d675c9420d","Type":"ContainerStarted","Data":"f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1"}
Nov 02 14:29:20.045772 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:20.045278    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz" event={"ID":"434a8e25-6856-46f2-b0ea-12d675c9420d","Type":"ContainerStarted","Data":"6f065a283cbc05f826e6e302d44b85fd7e3a5d5681e92b19f0a42bdb909bc3a6"}
Nov 02 14:29:20.045772 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:20.045548    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:20.085794 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:20.085736    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz" podStartSLOduration=2.085719962 podStartE2EDuration="2.085719962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:20.08237847 +0000 UTC m=+2692.041754087" watchObservedRunningTime="2026-11-02 14:29:20.085719962 +0000 UTC m=+2692.045095568"
Nov 02 14:29:28.958235 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:28.957679    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:29:28.961853 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:28.961824    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:29:30.049173 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:30.049143    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:31.056233 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:31.056193    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:29:31.135820 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:31.135785    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"]
Nov 02 14:29:36.136764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.136709    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m" podUID="31810e37-907c-481e-be84-c9399536b616" containerName="payload-processing" containerID="cri-o://0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523" gracePeriod=25
Nov 02 14:29:36.419255 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.419225    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:36.507253 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.507221    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-provider-credentials\") pod \"31810e37-907c-481e-be84-c9399536b616\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") "
Nov 02 14:29:36.507486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.507297    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-kube-api-access-rdwq6\" (UniqueName: \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-kube-api-access-rdwq6\") pod \"31810e37-907c-481e-be84-c9399536b616\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") "
Nov 02 14:29:36.507486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.507339    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-plugins-config-volume\") pod \"31810e37-907c-481e-be84-c9399536b616\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") "
Nov 02 14:29:36.507486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.507454    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-routing-overlay-volume\") pod \"31810e37-907c-481e-be84-c9399536b616\" (UID: \"31810e37-907c-481e-be84-c9399536b616\") "
Nov 02 14:29:36.507905 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.507875    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-plugins-config-volume" pod "31810e37-907c-481e-be84-c9399536b616" (UID: "31810e37-907c-481e-be84-c9399536b616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:36.507905 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.507887    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-routing-overlay-volume" pod "31810e37-907c-481e-be84-c9399536b616" (UID: "31810e37-907c-481e-be84-c9399536b616"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:36.508104 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.507993    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:36.508104 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.508007    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31810e37-907c-481e-be84-c9399536b616-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:36.510252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.510218    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-kube-api-access-rdwq6" pod "31810e37-907c-481e-be84-c9399536b616" (UID: "31810e37-907c-481e-be84-c9399536b616"). InnerVolumeSpecName "kube-api-access-rdwq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:36.510375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.510245    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-provider-credentials" pod "31810e37-907c-481e-be84-c9399536b616" (UID: "31810e37-907c-481e-be84-c9399536b616"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:36.616529 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.616487    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:36.616529 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:36.616531    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdwq6\" (UniqueName: \"kubernetes.io/projected/31810e37-907c-481e-be84-c9399536b616-kube-api-access-rdwq6\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:29:37.191347 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:37.191306    6284 generic.go:356] "Generic (PLEG): container finished" podID="31810e37-907c-481e-be84-c9399536b616" containerID="0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523" exitCode=0
Nov 02 14:29:37.191852 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:37.191356    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"
Nov 02 14:29:37.191852 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:37.191396    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m" event={"ID":"31810e37-907c-481e-be84-c9399536b616","Type":"ContainerDied","Data":"0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523"}
Nov 02 14:29:37.191852 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:37.191457    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m" event={"ID":"31810e37-907c-481e-be84-c9399536b616","Type":"ContainerDied","Data":"02f959d8d61a3a86ce5f0066bb573f974f2b68a451879e147b53b7320c0b820f"}
Nov 02 14:29:37.191852 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:37.191489    6284 scope.go:122] "RemoveContainer" containerID="0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523"
Nov 02 14:29:37.203037 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:37.203014    6284 scope.go:122] "RemoveContainer" containerID="0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523"
Nov 02 14:29:37.203352 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:29:37.203331    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523\": container with ID starting with 0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523 not found: ID does not exist" containerID="0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523"
Nov 02 14:29:37.203425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:37.203374    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523"} err="failed to get container status \"0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523\": rpc error: code = NotFound desc = could not find container \"0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523\": container with ID starting with 0a86780d70bf3ef7e10990d2f47c474623bc20d20ed80756a4b8a27caea20523 not found: ID does not exist"
Nov 02 14:29:37.237407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:37.237373    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"]
Nov 02 14:29:37.247975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:37.247925    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-6cc84vx8m"]
Nov 02 14:29:38.170329 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:38.170294    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31810e37-907c-481e-be84-c9399536b616" path="/var/lib/kubelet/pods/31810e37-907c-481e-be84-c9399536b616/volumes"
Nov 02 14:29:43.900073 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:29:43.900024    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:29:44.451781 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.451741    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"]
Nov 02 14:29:44.452435 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.452419    6284 cpu_manager.go:414] "containerMap: removing container" podUID="31810e37-907c-481e-be84-c9399536b616" containerName="payload-processing"
Nov 02 14:29:44.452481 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.452436    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31810e37-907c-481e-be84-c9399536b616" containerName="payload-processing"
Nov 02 14:29:44.452565 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.452556    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="31810e37-907c-481e-be84-c9399536b616" containerName="payload-processing"
Nov 02 14:29:44.456636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.456612    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.470567 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.470534    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"agc-cred-187d768557d1\""
Nov 02 14:29:44.480551 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.480514    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"payload-processing-external-model-e2e--1d26709d-dockercfg-v6xgj\""
Nov 02 14:29:44.480764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.480627    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2d13f4\""
Nov 02 14:29:44.480764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.480650    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"e2e-cfl-4239ef-provider-cred\""
Nov 02 14:29:44.480764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.480676    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"kube-root-ca.crt\""
Nov 02 14:29:44.480764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.480705    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"openshift-service-ca.crt\""
Nov 02 14:29:44.480925 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.480814    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"routing-overlay\""
Nov 02 14:29:44.484698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.484665    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"]
Nov 02 14:29:44.530217 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.530183    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.530388 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.530271    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.530388 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.530319    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.530467 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.530438    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdntz\" (UniqueName: \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-kube-api-access-cdntz\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.631691 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.631651    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdntz\" (UniqueName: \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-kube-api-access-cdntz\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.631906 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.631701    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.631906 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.631799    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.631906 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.631874    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.632641 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.632606    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.633003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.632982    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.634691 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.634667    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.685339 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.685263    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdntz\" (UniqueName: \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-kube-api-access-cdntz\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:44.772351 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:44.772057    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:45.095806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:45.095770    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"]
Nov 02 14:29:45.097411 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:29:45.097373    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode399e59a_a8ea_42a1_b3d2_d6135c78e5b2.slice/crio-753fb0f2adf22be3d32a44fea46d22dd6f794a33a39faa5dcb184a6229710163 WatchSource:0}: Error finding container 753fb0f2adf22be3d32a44fea46d22dd6f794a33a39faa5dcb184a6229710163: Status 404 returned error can't find the container with id 753fb0f2adf22be3d32a44fea46d22dd6f794a33a39faa5dcb184a6229710163
Nov 02 14:29:45.258395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:45.258352    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4" event={"ID":"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2","Type":"ContainerStarted","Data":"e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320"}
Nov 02 14:29:45.258600 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:45.258427    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4" event={"ID":"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2","Type":"ContainerStarted","Data":"753fb0f2adf22be3d32a44fea46d22dd6f794a33a39faa5dcb184a6229710163"}
Nov 02 14:29:45.258600 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:45.258454    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:29:45.299379 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:45.299311    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4" podStartSLOduration=1.299289174 podStartE2EDuration="1.299289174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:45.297434255 +0000 UTC m=+2717.256809862" watchObservedRunningTime="2026-11-02 14:29:45.299289174 +0000 UTC m=+2717.258664783"
Nov 02 14:29:56.295781 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:29:56.294084    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:30:09.957341 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:09.957299    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"]
Nov 02 14:30:11.814995 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:11.814958    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"]
Nov 02 14:30:11.819599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:11.819571    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:11.836576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:11.836541    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"agc-cred-3fbd4dc2e303\""
Nov 02 14:30:11.847624 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:11.847580    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"]
Nov 02 14:30:11.975835 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:11.975793    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:11.976073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:11.976051    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:11.976223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:11.976199    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:11.976263 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:11.976240    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xg8\" (UniqueName: \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-kube-api-access-f4xg8\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.082900 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.082793    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.084171 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.084144    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.085269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.085245    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.085427 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.085412    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xg8\" (UniqueName: \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-kube-api-access-f4xg8\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.085951 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.083923    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.086206 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:12.086186    6284 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d13f4/agc-cred-3fbd4dc2e303: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:30:12.086318 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:12.086307    6284 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:30:12.086472 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:12.086461    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials podName:1ffb1486-0d46-477f-84af-4dea4ae56864 nodeName:}" failed. No retries permitted until 2026-11-02 14:30:12.586438647 +0000 UTC m=+2744.545814247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7" (UID: "1ffb1486-0d46-477f-84af-4dea4ae56864") : secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:30:12.086753 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.085098    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.109032 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.109004    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xg8\" (UniqueName: \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-kube-api-access-f4xg8\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.334250 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.334162    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"]
Nov 02 14:30:12.334573 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:12.334555    6284 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7" podUID="1ffb1486-0d46-477f-84af-4dea4ae56864"
Nov 02 14:30:12.498012 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.497923    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.512624 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.512587    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.591252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.591149    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-plugins-config-volume\") pod \"1ffb1486-0d46-477f-84af-4dea4ae56864\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") "
Nov 02 14:30:12.591433 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.591382    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-routing-overlay-volume\") pod \"1ffb1486-0d46-477f-84af-4dea4ae56864\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") "
Nov 02 14:30:12.591518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.591444    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-kube-api-access-f4xg8\" (UniqueName: \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-kube-api-access-f4xg8\") pod \"1ffb1486-0d46-477f-84af-4dea4ae56864\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") "
Nov 02 14:30:12.591676 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.591579    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-plugins-config-volume" pod "1ffb1486-0d46-477f-84af-4dea4ae56864" (UID: "1ffb1486-0d46-477f-84af-4dea4ae56864"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:12.591791 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.591770    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-routing-overlay-volume" pod "1ffb1486-0d46-477f-84af-4dea4ae56864" (UID: "1ffb1486-0d46-477f-84af-4dea4ae56864"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:12.591870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.591851    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:12.592055 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.592041    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:12.592124 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.592056    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffb1486-0d46-477f-84af-4dea4ae56864-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:12.592124 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:12.592060    6284 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d13f4/agc-cred-3fbd4dc2e303: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:30:12.592124 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:12.592077    6284 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:30:12.592275 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:12.592129    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials podName:1ffb1486-0d46-477f-84af-4dea4ae56864 nodeName:}" failed. No retries permitted until 2026-11-02 14:30:13.592111078 +0000 UTC m=+2745.551486668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7" (UID: "1ffb1486-0d46-477f-84af-4dea4ae56864") : secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:30:12.593623 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.593598    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-kube-api-access-f4xg8" pod "1ffb1486-0d46-477f-84af-4dea4ae56864" (UID: "1ffb1486-0d46-477f-84af-4dea4ae56864"). InnerVolumeSpecName "kube-api-access-f4xg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:12.693657 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:12.693619    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xg8\" (UniqueName: \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-kube-api-access-f4xg8\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:13.504054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:13.504022    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:13.579538 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:13.579492    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"]
Nov 02 14:30:13.591606 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:13.591561    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"]
Nov 02 14:30:13.607783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:13.607727    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7\" (UID: \"1ffb1486-0d46-477f-84af-4dea4ae56864\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7"
Nov 02 14:30:13.608024 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:13.607996    6284 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d13f4/agc-cred-3fbd4dc2e303: object "ai-tenant-e2e-extmod-a-2d13f4"/"agc-cred-3fbd4dc2e303" not registered
Nov 02 14:30:13.608076 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:13.608028    6284 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7: object "ai-tenant-e2e-extmod-a-2d13f4"/"agc-cred-3fbd4dc2e303" not registered
Nov 02 14:30:13.608137 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:13.608124    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials podName:1ffb1486-0d46-477f-84af-4dea4ae56864 nodeName:}" failed. No retries permitted until 2026-11-02 14:30:15.608100182 +0000 UTC m=+2747.567475771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d13f4-765fg9hc7" (UID: "1ffb1486-0d46-477f-84af-4dea4ae56864") : object "ai-tenant-e2e-extmod-a-2d13f4"/"agc-cred-3fbd4dc2e303" not registered
Nov 02 14:30:13.709078 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:13.709029    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1ffb1486-0d46-477f-84af-4dea4ae56864-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:13.896710 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:13.896603    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:30:14.171620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:14.171529    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffb1486-0d46-477f-84af-4dea4ae56864" path="/var/lib/kubelet/pods/1ffb1486-0d46-477f-84af-4dea4ae56864/volumes"
Nov 02 14:30:14.958415 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:14.958354    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4" podUID="e399e59a-a8ea-42a1-b3d2-d6135c78e5b2" containerName="payload-processing" containerID="cri-o://e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320" gracePeriod=25
Nov 02 14:30:15.242665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.242637    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:30:15.329507 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.329458    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-routing-overlay-volume\") pod \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") "
Nov 02 14:30:15.329751 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.329562    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-provider-credentials\") pod \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") "
Nov 02 14:30:15.329751 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.329693    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-kube-api-access-cdntz\" (UniqueName: \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-kube-api-access-cdntz\") pod \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") "
Nov 02 14:30:15.329989 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.329778    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-plugins-config-volume\") pod \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\" (UID: \"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2\") "
Nov 02 14:30:15.329989 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.329929    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-routing-overlay-volume" pod "e399e59a-a8ea-42a1-b3d2-d6135c78e5b2" (UID: "e399e59a-a8ea-42a1-b3d2-d6135c78e5b2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:15.330411 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.330386    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-plugins-config-volume" pod "e399e59a-a8ea-42a1-b3d2-d6135c78e5b2" (UID: "e399e59a-a8ea-42a1-b3d2-d6135c78e5b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:15.330545 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.330467    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:15.332097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.332063    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-kube-api-access-cdntz" pod "e399e59a-a8ea-42a1-b3d2-d6135c78e5b2" (UID: "e399e59a-a8ea-42a1-b3d2-d6135c78e5b2"). InnerVolumeSpecName "kube-api-access-cdntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:15.332204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.332173    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-provider-credentials" pod "e399e59a-a8ea-42a1-b3d2-d6135c78e5b2" (UID: "e399e59a-a8ea-42a1-b3d2-d6135c78e5b2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:15.431900 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.431860    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdntz\" (UniqueName: \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-kube-api-access-cdntz\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:15.431900 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.431896    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:15.431900 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.431906    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:15.520248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.520207    6284 generic.go:356] "Generic (PLEG): container finished" podID="e399e59a-a8ea-42a1-b3d2-d6135c78e5b2" containerID="e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320" exitCode=0
Nov 02 14:30:15.520518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.520259    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"
Nov 02 14:30:15.520518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.520300    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4" event={"ID":"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2","Type":"ContainerDied","Data":"e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320"}
Nov 02 14:30:15.520518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.520369    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4" event={"ID":"e399e59a-a8ea-42a1-b3d2-d6135c78e5b2","Type":"ContainerDied","Data":"753fb0f2adf22be3d32a44fea46d22dd6f794a33a39faa5dcb184a6229710163"}
Nov 02 14:30:15.520518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.520398    6284 scope.go:122] "RemoveContainer" containerID="e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320"
Nov 02 14:30:15.538014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.537994    6284 scope.go:122] "RemoveContainer" containerID="e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320"
Nov 02 14:30:15.538382 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:15.538360    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320\": container with ID starting with e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320 not found: ID does not exist" containerID="e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320"
Nov 02 14:30:15.538434 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.538394    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320"} err="failed to get container status \"e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320\": rpc error: code = NotFound desc = could not find container \"e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320\": container with ID starting with e1bc2548497c5631ffbbbee783668afa3f9f05b232753a3cff0e0e493150c320 not found: ID does not exist"
Nov 02 14:30:15.570216 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.570174    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"]
Nov 02 14:30:15.579681 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:15.579639    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-86d66n8n4"]
Nov 02 14:30:16.170767 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:16.170737    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e399e59a-a8ea-42a1-b3d2-d6135c78e5b2" path="/var/lib/kubelet/pods/e399e59a-a8ea-42a1-b3d2-d6135c78e5b2/volumes"
Nov 02 14:30:23.637379 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.637340    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"]
Nov 02 14:30:23.638087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.638055    6284 cpu_manager.go:414] "containerMap: removing container" podUID="e399e59a-a8ea-42a1-b3d2-d6135c78e5b2" containerName="payload-processing"
Nov 02 14:30:23.638087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.638082    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e399e59a-a8ea-42a1-b3d2-d6135c78e5b2" containerName="payload-processing"
Nov 02 14:30:23.638318 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.638303    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="e399e59a-a8ea-42a1-b3d2-d6135c78e5b2" containerName="payload-processing"
Nov 02 14:30:23.644067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.644040    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.666659 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.666621    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"openshift-service-ca.crt\""
Nov 02 14:30:23.666856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.666625    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"routing-overlay\""
Nov 02 14:30:23.666856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.666625    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"agc-cred-1665ff760a19\""
Nov 02 14:30:23.666856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.666628    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"agc-cred-85e62aa25307\""
Nov 02 14:30:23.666856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.666720    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"kube-root-ca.crt\""
Nov 02 14:30:23.666856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.666751    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"payload-processing-external-model-plugins-e2e-extmod-a-26b508\""
Nov 02 14:30:23.667133 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.667012    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"payload-processing-external-model-e2e--7e93d834-dockercfg-vv77g\""
Nov 02 14:30:23.672062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.672018    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"]
Nov 02 14:30:23.697931 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.697878    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.698348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.698315    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.698437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.698382    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.698529 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.698510    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhnmq\" (UniqueName: \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-kube-api-access-hhnmq\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.799584 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.799542    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.799817 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.799597    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.799817 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.799675    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnmq\" (UniqueName: \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-kube-api-access-hhnmq\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.799817 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.799769    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.800403 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.800375    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.800929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.800908    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.802579 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.802558    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.821798 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.821761    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhnmq\" (UniqueName: \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-kube-api-access-hhnmq\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:23.964723 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:23.964628    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:24.282752 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:24.282722    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"]
Nov 02 14:30:24.285719 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:30:24.285687    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a82f19d_667e_4efe_95e0_2ebf6e2e88a6.slice/crio-dd1e76e77c419c16d34468447aa9560eb9b1c21a7f2d341e212b0a6fb77b883f WatchSource:0}: Error finding container dd1e76e77c419c16d34468447aa9560eb9b1c21a7f2d341e212b0a6fb77b883f: Status 404 returned error can't find the container with id dd1e76e77c419c16d34468447aa9560eb9b1c21a7f2d341e212b0a6fb77b883f
Nov 02 14:30:24.608611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:24.608510    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk" event={"ID":"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6","Type":"ContainerStarted","Data":"21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60"}
Nov 02 14:30:24.608611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:24.608575    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk" event={"ID":"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6","Type":"ContainerStarted","Data":"dd1e76e77c419c16d34468447aa9560eb9b1c21a7f2d341e212b0a6fb77b883f"}
Nov 02 14:30:24.608814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:24.608757    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:24.658723 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:24.658664    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk" podStartSLOduration=1.658643716 podStartE2EDuration="1.658643716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:24.655589694 +0000 UTC m=+2756.614965302" watchObservedRunningTime="2026-11-02 14:30:24.658643716 +0000 UTC m=+2756.618019323"
Nov 02 14:30:35.618594 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:35.618561    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:43.895990 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:43.895929    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:30:46.978550 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:46.978512    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"]
Nov 02 14:30:51.980006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:51.979885    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk" podUID="5a82f19d-667e-4efe-95e0-2ebf6e2e88a6" containerName="payload-processing" containerID="cri-o://21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60" gracePeriod=25
Nov 02 14:30:52.231762 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.231671    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"]
Nov 02 14:30:52.237679 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.237650    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.258193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.258132    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:52.264845 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.264817    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"]
Nov 02 14:30:52.278583 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.278540    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"]
Nov 02 14:30:52.279445 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.279418    6284 cpu_manager.go:414] "containerMap: removing container" podUID="5a82f19d-667e-4efe-95e0-2ebf6e2e88a6" containerName="payload-processing"
Nov 02 14:30:52.279779 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.279756    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a82f19d-667e-4efe-95e0-2ebf6e2e88a6" containerName="payload-processing"
Nov 02 14:30:52.280342 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.280317    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a82f19d-667e-4efe-95e0-2ebf6e2e88a6" containerName="payload-processing"
Nov 02 14:30:52.292829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.292792    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.309298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.308257    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"]
Nov 02 14:30:52.395125 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395086    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-routing-overlay-volume\") pod \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") "
Nov 02 14:30:52.395317 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395190    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-kube-api-access-hhnmq\" (UniqueName: \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-kube-api-access-hhnmq\") pod \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") "
Nov 02 14:30:52.395317 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395284    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-plugins-config-volume\") pod \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") "
Nov 02 14:30:52.395421 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395399    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-provider-credentials\") pod \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\" (UID: \"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6\") "
Nov 02 14:30:52.395549 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395525    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.395624 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395562    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-routing-overlay-volume" pod "5a82f19d-667e-4efe-95e0-2ebf6e2e88a6" (UID: "5a82f19d-667e-4efe-95e0-2ebf6e2e88a6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:52.395624 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395574    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx625\" (UniqueName: \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-kube-api-access-gx625\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.395720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395698    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-plugins-config-volume" pod "5a82f19d-667e-4efe-95e0-2ebf6e2e88a6" (UID: "5a82f19d-667e-4efe-95e0-2ebf6e2e88a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:52.395720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395703    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.395803 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395766    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.395863 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395835    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.396012 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.395986    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.396080 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396033    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56655b89-7919-4694-8933-fbcc61b81de0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.396122 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396101    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.396162 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396129    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.396222 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396209    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.396260 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396236    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.396304 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396284    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56655b89-7919-4694-8933-fbcc61b81de0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.396376 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396364    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.396414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396393    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.396452 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396418    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78tww\" (UniqueName: \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-kube-api-access-78tww\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.396490 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396457    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.396529 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396511    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.396580 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396555    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.396649 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396636    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:52.396697 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.396681    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:52.397695 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.397672    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-kube-api-access-hhnmq" pod "5a82f19d-667e-4efe-95e0-2ebf6e2e88a6" (UID: "5a82f19d-667e-4efe-95e0-2ebf6e2e88a6"). InnerVolumeSpecName "kube-api-access-hhnmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:52.397784 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.397713    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-provider-credentials" pod "5a82f19d-667e-4efe-95e0-2ebf6e2e88a6" (UID: "5a82f19d-667e-4efe-95e0-2ebf6e2e88a6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:52.498349 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498305    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.498349 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498352    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56655b89-7919-4694-8933-fbcc61b81de0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.498645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498387    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.498645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498410    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.498645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498456    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.498645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498491    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.498645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498546    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56655b89-7919-4694-8933-fbcc61b81de0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.498645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498606    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.498645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498643    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.499081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498680    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78tww\" (UniqueName: \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-kube-api-access-78tww\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.499081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498712    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.499081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498755    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.499081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498786    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.499081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498828    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.499081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498897    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx625\" (UniqueName: \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-kube-api-access-gx625\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.499081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498909    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.499081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498955    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.499081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.498991    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.499081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.499046    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.499602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.499099    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.499602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.499123    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.499602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.499346    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56655b89-7919-4694-8933-fbcc61b81de0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.499602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.499437    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.499602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.499478    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.499602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.499551    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.499994 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.499761    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.499994 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.499824    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:52.499994 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.499847    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhnmq\" (UniqueName: \"kubernetes.io/projected/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6-kube-api-access-hhnmq\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:30:52.500148 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.500076    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.500148 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.500122    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.501202 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.501172    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.501342 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.501324    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56655b89-7919-4694-8933-fbcc61b81de0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.502051 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.502030    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.502253 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.502240    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.520206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.520171    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.520342 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.520171    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.521185 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.521161    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx625\" (UniqueName: \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-kube-api-access-gx625\") pod \"maas-default-gateway-openshift-default-6886b96f9-226kc\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.521294 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.521249    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78tww\" (UniqueName: \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-kube-api-access-78tww\") pod \"maas-default-gateway-openshift-default-6886b96f9-c9cq8\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.594295 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.594256    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:52.626516 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.626472    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:52.902746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.902664    6284 generic.go:356] "Generic (PLEG): container finished" podID="5a82f19d-667e-4efe-95e0-2ebf6e2e88a6" containerID="21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60" exitCode=0
Nov 02 14:30:52.902746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.902710    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"
Nov 02 14:30:52.903060 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.902746    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk" event={"ID":"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6","Type":"ContainerDied","Data":"21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60"}
Nov 02 14:30:52.903060 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.902812    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk" event={"ID":"5a82f19d-667e-4efe-95e0-2ebf6e2e88a6","Type":"ContainerDied","Data":"dd1e76e77c419c16d34468447aa9560eb9b1c21a7f2d341e212b0a6fb77b883f"}
Nov 02 14:30:52.903060 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.902844    6284 scope.go:122] "RemoveContainer" containerID="21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60"
Nov 02 14:30:52.914508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.914485    6284 scope.go:122] "RemoveContainer" containerID="21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60"
Nov 02 14:30:52.914895 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:30:52.914870    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60\": container with ID starting with 21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60 not found: ID does not exist" containerID="21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60"
Nov 02 14:30:52.914994 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.914905    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60"} err="failed to get container status \"21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60\": rpc error: code = NotFound desc = could not find container \"21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60\": container with ID starting with 21381858edf85c6f52b9dda42d7a14177f82258b46f07f553dd465ab9886aa60 not found: ID does not exist"
Nov 02 14:30:52.922025 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.921966    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"]
Nov 02 14:30:52.923717 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:30:52.923687    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56655b89_7919_4694_8933_fbcc61b81de0.slice/crio-c49b359bf7eea32eda820ad203d3bd176e90cbebb6e81d4ab57aa1536b72a429 WatchSource:0}: Error finding container c49b359bf7eea32eda820ad203d3bd176e90cbebb6e81d4ab57aa1536b72a429: Status 404 returned error can't find the container with id c49b359bf7eea32eda820ad203d3bd176e90cbebb6e81d4ab57aa1536b72a429
Nov 02 14:30:52.937929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.937810    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:30:52.938203 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.938092    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:30:52.938549 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.938497    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:30:52.967173 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.967139    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"]
Nov 02 14:30:52.970718 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:30:52.970683    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68a59a0_b4cf_40e0_9f58_4f0db1436596.slice/crio-7c52a1a6e697425facfea02596bffec61bb7349e5895318edef5a0ea9d4d384d WatchSource:0}: Error finding container 7c52a1a6e697425facfea02596bffec61bb7349e5895318edef5a0ea9d4d384d: Status 404 returned error can't find the container with id 7c52a1a6e697425facfea02596bffec61bb7349e5895318edef5a0ea9d4d384d
Nov 02 14:30:52.971818 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.971689    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"]
Nov 02 14:30:52.974334 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.974300    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:30:52.974446 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.974371    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:30:52.974446 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.974400    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:30:52.975493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:52.975460    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-7486k5bjk"]
Nov 02 14:30:53.915592 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:53.915555    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" event={"ID":"56655b89-7919-4694-8933-fbcc61b81de0","Type":"ContainerStarted","Data":"309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4"}
Nov 02 14:30:53.916155 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:53.915618    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" event={"ID":"56655b89-7919-4694-8933-fbcc61b81de0","Type":"ContainerStarted","Data":"c49b359bf7eea32eda820ad203d3bd176e90cbebb6e81d4ab57aa1536b72a429"}
Nov 02 14:30:53.917377 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:53.917344    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" event={"ID":"e68a59a0-b4cf-40e0-9f58-4f0db1436596","Type":"ContainerStarted","Data":"384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d"}
Nov 02 14:30:53.917502 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:53.917385    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" event={"ID":"e68a59a0-b4cf-40e0-9f58-4f0db1436596","Type":"ContainerStarted","Data":"7c52a1a6e697425facfea02596bffec61bb7349e5895318edef5a0ea9d4d384d"}
Nov 02 14:30:53.986456 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:53.986376    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" podStartSLOduration=1.986355307 podStartE2EDuration="1.986355307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:53.983400432 +0000 UTC m=+2785.942776040" watchObservedRunningTime="2026-11-02 14:30:53.986355307 +0000 UTC m=+2785.945730912"
Nov 02 14:30:54.208530 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:54.208445    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a82f19d-667e-4efe-95e0-2ebf6e2e88a6" path="/var/lib/kubelet/pods/5a82f19d-667e-4efe-95e0-2ebf6e2e88a6/volumes"
Nov 02 14:30:54.599963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:54.597851    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:30:54.600765 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:54.600293    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-226kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 14:30:54.600765 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:54.600378    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 14:30:54.626915 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:54.626878    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:30:54.628444 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:54.628417    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c9cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 14:30:54.628554 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:54.628475    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 14:30:55.603102 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:55.601565    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-226kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 14:30:55.603102 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:55.601636    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 14:30:55.628957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:55.628337    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c9cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 14:30:55.628957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:55.628403    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 14:30:56.602394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.602315    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-226kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 14:30:56.602394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.602385    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 14:30:56.605208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.605140    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" podStartSLOduration=4.605114555 podStartE2EDuration="4.605114555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:54.048740499 +0000 UTC m=+2786.008116109" watchObservedRunningTime="2026-11-02 14:30:56.605114555 +0000 UTC m=+2788.564490155"
Nov 02 14:30:56.611006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.606648    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"]
Nov 02 14:30:56.616610 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.616563    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"]
Nov 02 14:30:56.616820 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.616690    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.627042 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.627012    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c9cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 14:30:56.627155 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.627071    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 14:30:56.630322 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.630289    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 14:30:56.639948 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.639906    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-865bae346716\""
Nov 02 14:30:56.640138 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.639976    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f8ae8e27fe78\""
Nov 02 14:30:56.640138 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.639909    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 14:30:56.640138 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.639913    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 14:30:56.640138 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.640066    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 14:30:56.640138 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.640072    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-glvff\""
Nov 02 14:30:56.801004 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.800893    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-routing-overlay-volume\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.802119 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.801428    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-992qd\" (UniqueName: \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-kube-api-access-992qd\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.802395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.802286    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-plugins-config-volume\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.802395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.802326    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-provider-credentials\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.936022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.926158    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-routing-overlay-volume\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.936022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.926253    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992qd\" (UniqueName: \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-kube-api-access-992qd\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.936022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.926386    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-plugins-config-volume\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.936022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.926411    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-provider-credentials\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.936022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.928083    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-routing-overlay-volume\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.936022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.928818    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-plugins-config-volume\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.963963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.942325    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-provider-credentials\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:56.991418 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:56.986503    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-992qd\" (UniqueName: \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-kube-api-access-992qd\") pod \"payload-processing-external-model-56f5855db-l7f7j\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") " pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:57.233391 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:57.233363    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:57.625054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:57.621689    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-226kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 14:30:57.625054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:57.621833    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 14:30:57.632038 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:57.631995    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c9cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 14:30:57.632222 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:57.632078    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 14:30:57.687103 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:57.687058    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"]
Nov 02 14:30:58.017364 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:58.017323    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j" event={"ID":"2e46e749-9ab2-48b1-b59e-872aef58c986","Type":"ContainerStarted","Data":"2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d"}
Nov 02 14:30:58.017572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:58.017395    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j" event={"ID":"2e46e749-9ab2-48b1-b59e-872aef58c986","Type":"ContainerStarted","Data":"3044841693c94fe830871d195887f976e864317b15a6970c400b707d973d47aa"}
Nov 02 14:30:58.018095 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:58.018069    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:30:58.077963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:58.076908    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j" podStartSLOduration=2.076884846 podStartE2EDuration="2.076884846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:58.058192526 +0000 UTC m=+2790.017568135" watchObservedRunningTime="2026-11-02 14:30:58.076884846 +0000 UTC m=+2790.036260454"
Nov 02 14:30:58.601963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:58.599379    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-226kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 14:30:58.601963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:58.599456    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 14:30:58.627079 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:58.627041    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c9cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 14:30:58.627472 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:58.627092    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 14:30:59.594664 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:59.594624    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-226kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 14:30:59.594879 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:59.594698    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 14:30:59.627425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:59.627392    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c9cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 14:30:59.627811 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:30:59.627451    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 14:31:00.605864 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:00.605366    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-226kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 14:31:00.605864 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:00.605425    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 14:31:00.633901 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:00.633866    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c9cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 14:31:00.634587 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:00.634539    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 14:31:02.632510 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:02.632473    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-c9cq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:31:02.633109 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:02.633077    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:31:02.643141 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:02.643105    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-226kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:31:02.643395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:02.643364    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:31:02.712903 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:02.712867    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:31:02.714282 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:02.714252    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:31:02.717318 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:02.717282    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:31:02.756373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:02.756341    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:31:02.756709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:02.756686    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:31:02.757711 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:02.757692    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:31:05.849791 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.849755    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"]
Nov 02 14:31:05.854454 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.854428    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:05.903111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.903077    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"kube-root-ca.crt\""
Nov 02 14:31:05.903111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.903077    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"routing-overlay\""
Nov 02 14:31:05.903111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.903089    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"payload-processing-external-model-plugins-e2e-extmod-a-26b508\""
Nov 02 14:31:05.903458 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.903132    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"openshift-service-ca.crt\""
Nov 02 14:31:05.903458 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.903154    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"agc-cred-a6650dd4fae6\""
Nov 02 14:31:05.903458 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.903210    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-26b508\"/\"payload-processing-external-model-e2e--7e93d834-dockercfg-gcw7q\""
Nov 02 14:31:05.907514 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.907477    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"]
Nov 02 14:31:05.984351 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.984308    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:05.984556 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.984518    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xnsd\" (UniqueName: \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-kube-api-access-7xnsd\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:05.984605 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.984561    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:05.984766 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:05.984740    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:06.085779 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:06.085739    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:06.086063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:06.085797    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:06.086063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:06.086024    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xnsd\" (UniqueName: \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-kube-api-access-7xnsd\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:06.086193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:06.086087    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:06.086808 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:06.086777    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:06.086917 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:06.086884    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:06.088536 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:06.088506    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:06.107400 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:06.107288    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xnsd\" (UniqueName: \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-kube-api-access-7xnsd\") pod \"payload-processing-external-model-e2e-extmod-a-26b508-8457r684j\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") " pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:06.177433 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:06.177391    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:06.567512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:06.567480    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"]
Nov 02 14:31:06.569517 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:31:06.569484    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb8f4c6_be95_427e_b4a4_59e3d6b56e46.slice/crio-8acd97eeed13273d4cd1ff2198ef7d12b42a943739e41c537b6ba143d097c868 WatchSource:0}: Error finding container 8acd97eeed13273d4cd1ff2198ef7d12b42a943739e41c537b6ba143d097c868: Status 404 returned error can't find the container with id 8acd97eeed13273d4cd1ff2198ef7d12b42a943739e41c537b6ba143d097c868
Nov 02 14:31:07.178019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:07.177980    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j" event={"ID":"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46","Type":"ContainerStarted","Data":"9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab"}
Nov 02 14:31:07.178534 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:07.178038    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j" event={"ID":"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46","Type":"ContainerStarted","Data":"8acd97eeed13273d4cd1ff2198ef7d12b42a943739e41c537b6ba143d097c868"}
Nov 02 14:31:07.178534 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:07.178091    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:07.221024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:07.218573    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j" podStartSLOduration=2.218559725 podStartE2EDuration="2.218559725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:31:07.218177145 +0000 UTC m=+2799.177552753" watchObservedRunningTime="2026-11-02 14:31:07.218559725 +0000 UTC m=+2799.177935332"
Nov 02 14:31:09.029521 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:09.029481    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:31:13.946862 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:31:13.946820    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:31:18.189106 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:31:18.189068    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:31:43.896058 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:31:43.896018    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:32:13.825199 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:13.825160    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"]
Nov 02 14:32:13.895322 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:32:13.895280    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:32:18.826253 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:18.826166    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz" podUID="434a8e25-6856-46f2-b0ea-12d675c9420d" containerName="payload-processing" containerID="cri-o://f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1" gracePeriod=25
Nov 02 14:32:19.104478 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.104448    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:32:19.203348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.203314    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-kube-api-access-q4b85\" (UniqueName: \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-kube-api-access-q4b85\") pod \"434a8e25-6856-46f2-b0ea-12d675c9420d\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") "
Nov 02 14:32:19.203348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.203355    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-plugins-config-volume\") pod \"434a8e25-6856-46f2-b0ea-12d675c9420d\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") "
Nov 02 14:32:19.203644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.203444    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-provider-credentials\") pod \"434a8e25-6856-46f2-b0ea-12d675c9420d\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") "
Nov 02 14:32:19.203644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.203478    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-routing-overlay-volume\") pod \"434a8e25-6856-46f2-b0ea-12d675c9420d\" (UID: \"434a8e25-6856-46f2-b0ea-12d675c9420d\") "
Nov 02 14:32:19.203927 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.203897    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-plugins-config-volume" pod "434a8e25-6856-46f2-b0ea-12d675c9420d" (UID: "434a8e25-6856-46f2-b0ea-12d675c9420d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:32:19.204070 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.203992    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-routing-overlay-volume" pod "434a8e25-6856-46f2-b0ea-12d675c9420d" (UID: "434a8e25-6856-46f2-b0ea-12d675c9420d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:32:19.204514 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.204489    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:32:19.204594 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.204519    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/434a8e25-6856-46f2-b0ea-12d675c9420d-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:32:19.205806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.205783    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-provider-credentials" pod "434a8e25-6856-46f2-b0ea-12d675c9420d" (UID: "434a8e25-6856-46f2-b0ea-12d675c9420d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:32:19.205806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.205795    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-kube-api-access-q4b85" pod "434a8e25-6856-46f2-b0ea-12d675c9420d" (UID: "434a8e25-6856-46f2-b0ea-12d675c9420d"). InnerVolumeSpecName "kube-api-access-q4b85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:32:19.305994 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.305954    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:32:19.305994 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.305988    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4b85\" (UniqueName: \"kubernetes.io/projected/434a8e25-6856-46f2-b0ea-12d675c9420d-kube-api-access-q4b85\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:32:19.852451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.852412    6284 generic.go:356] "Generic (PLEG): container finished" podID="434a8e25-6856-46f2-b0ea-12d675c9420d" containerID="f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1" exitCode=0
Nov 02 14:32:19.852866 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.852461    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"
Nov 02 14:32:19.852866 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.852508    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz" event={"ID":"434a8e25-6856-46f2-b0ea-12d675c9420d","Type":"ContainerDied","Data":"f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1"}
Nov 02 14:32:19.852866 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.852566    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz" event={"ID":"434a8e25-6856-46f2-b0ea-12d675c9420d","Type":"ContainerDied","Data":"6f065a283cbc05f826e6e302d44b85fd7e3a5d5681e92b19f0a42bdb909bc3a6"}
Nov 02 14:32:19.852866 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.852594    6284 scope.go:122] "RemoveContainer" containerID="f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1"
Nov 02 14:32:19.868652 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.868623    6284 scope.go:122] "RemoveContainer" containerID="f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1"
Nov 02 14:32:19.869069 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:32:19.869044    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1\": container with ID starting with f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1 not found: ID does not exist" containerID="f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1"
Nov 02 14:32:19.869149 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.869082    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1"} err="failed to get container status \"f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1\": rpc error: code = NotFound desc = could not find container \"f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1\": container with ID starting with f22d83e50ff096f512a9a609db453f0deb2f08c2c23649055f35c5d7998439b1 not found: ID does not exist"
Nov 02 14:32:19.928531 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.928480    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"]
Nov 02 14:32:19.947713 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:19.947662    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-5f98a6/payload-processing-external-model-e2e-extmod-a-5f98a6-75cbk7lmz"]
Nov 02 14:32:20.172920 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:32:20.172841    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="434a8e25-6856-46f2-b0ea-12d675c9420d" path="/var/lib/kubelet/pods/434a8e25-6856-46f2-b0ea-12d675c9420d/volumes"
Nov 02 14:32:43.910995 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:32:43.910922    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:33:13.911076 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:33:13.910920    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:33:18.804852 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.804807    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"]
Nov 02 14:33:18.805492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.805475    6284 cpu_manager.go:414] "containerMap: removing container" podUID="434a8e25-6856-46f2-b0ea-12d675c9420d" containerName="payload-processing"
Nov 02 14:33:18.805492 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.805491    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434a8e25-6856-46f2-b0ea-12d675c9420d" containerName="payload-processing"
Nov 02 14:33:18.805613 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.805604    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="434a8e25-6856-46f2-b0ea-12d675c9420d" containerName="payload-processing"
Nov 02 14:33:18.809326 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.809303    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:33:18.851604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.851562    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"]
Nov 02 14:33:18.856952 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.856906    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-7bfb1f\"/\"e2e-usage-sim-99ce8c-cert\""
Nov 02 14:33:18.857175 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.857154    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-7bfb1f\"/\"default-dockercfg-q6q7l\""
Nov 02 14:33:18.858359 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.858329    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-7bfb1f\"/\"openshift-service-ca.crt\""
Nov 02 14:33:18.858512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.858393    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-7bfb1f\"/\"kube-root-ca.crt\""
Nov 02 14:33:18.911175 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.911139    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cb5658b0-11e9-402f-8685-1419b07b20a8-tls\") pod \"e2e-usage-sim-99ce8c-769cb9559f-87qmp\" (UID: \"cb5658b0-11e9-402f-8685-1419b07b20a8\") " pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:33:18.911358 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:18.911314    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hz9n\" (UniqueName: \"kubernetes.io/projected/cb5658b0-11e9-402f-8685-1419b07b20a8-kube-api-access-4hz9n\") pod \"e2e-usage-sim-99ce8c-769cb9559f-87qmp\" (UID: \"cb5658b0-11e9-402f-8685-1419b07b20a8\") " pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:33:19.013345 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.013309    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cb5658b0-11e9-402f-8685-1419b07b20a8-tls\") pod \"e2e-usage-sim-99ce8c-769cb9559f-87qmp\" (UID: \"cb5658b0-11e9-402f-8685-1419b07b20a8\") " pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:33:19.013538 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.013427    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hz9n\" (UniqueName: \"kubernetes.io/projected/cb5658b0-11e9-402f-8685-1419b07b20a8-kube-api-access-4hz9n\") pod \"e2e-usage-sim-99ce8c-769cb9559f-87qmp\" (UID: \"cb5658b0-11e9-402f-8685-1419b07b20a8\") " pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:33:19.015840 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.015806    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cb5658b0-11e9-402f-8685-1419b07b20a8-tls\") pod \"e2e-usage-sim-99ce8c-769cb9559f-87qmp\" (UID: \"cb5658b0-11e9-402f-8685-1419b07b20a8\") " pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:33:19.034811 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.034776    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hz9n\" (UniqueName: \"kubernetes.io/projected/cb5658b0-11e9-402f-8685-1419b07b20a8-kube-api-access-4hz9n\") pod \"e2e-usage-sim-99ce8c-769cb9559f-87qmp\" (UID: \"cb5658b0-11e9-402f-8685-1419b07b20a8\") " pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:33:19.123218 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.123126    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:33:19.431929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.431894    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"]
Nov 02 14:33:19.435008 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:33:19.434924    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb5658b0_11e9_402f_8685_1419b07b20a8.slice/crio-ba15c10dce7de22e48897c941e8d08b7263141a4507448569be467ce722b2541 WatchSource:0}: Error finding container ba15c10dce7de22e48897c941e8d08b7263141a4507448569be467ce722b2541: Status 404 returned error can't find the container with id ba15c10dce7de22e48897c941e8d08b7263141a4507448569be467ce722b2541
Nov 02 14:33:19.607522 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.607481    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp" event={"ID":"cb5658b0-11e9-402f-8685-1419b07b20a8","Type":"ContainerStarted","Data":"a632d16757520a07f6329ea21e911b8358a31772d1ebd075ace2bc786be9ca84"}
Nov 02 14:33:19.607705 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.607531    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp" event={"ID":"cb5658b0-11e9-402f-8685-1419b07b20a8","Type":"ContainerStarted","Data":"ba15c10dce7de22e48897c941e8d08b7263141a4507448569be467ce722b2541"}
Nov 02 14:33:19.607705 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.607575    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:33:19.609483 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.609450    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp" podUID="cb5658b0-11e9-402f-8685-1419b07b20a8" containerName="sim" probeResult="failure" output="Get \"https://10.134.0.145:8000/ready\": dial tcp 10.134.0.145:8000: connect: connection refused"
Nov 02 14:33:19.646409 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:19.646340    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp" podStartSLOduration=1.646320139 podStartE2EDuration="1.646320139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:19.64228007 +0000 UTC m=+2931.601655689" watchObservedRunningTime="2026-11-02 14:33:19.646320139 +0000 UTC m=+2931.605695780"
Nov 02 14:33:20.618833 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:20.618790    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:33:23.373642 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.373598    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"]
Nov 02 14:33:23.378479 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.378451    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.393144 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.393110    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a02c35-openshift-default-dockercfg-bm9cg\""
Nov 02 14:33:23.410578 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.408717    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"]
Nov 02 14:33:23.493004 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.492958    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-envoy\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.493206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.493015    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee52503f-b72e-42fe-a5b2-0a287941daca-istiod-ca-cert\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.493251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.493215    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.493298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.493265    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-data\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.493404 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.493388    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-podinfo\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.493468 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.493420    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-token\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.493468 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.493454    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-certs\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.493620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.493604    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-credential-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.493669 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.493662    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xbl\" (UniqueName: \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-kube-api-access-b9xbl\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.594680 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.594637    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.594907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.594705    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-data\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.594907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.594753    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-podinfo\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.594907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.594778    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-token\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.594907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.594807    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-certs\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.594907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.594898    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-credential-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.595195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.594987    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xbl\" (UniqueName: \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-kube-api-access-b9xbl\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.595195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.595079    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-envoy\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.595195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.595109    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee52503f-b72e-42fe-a5b2-0a287941daca-istiod-ca-cert\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.595195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.595132    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.595372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.595270    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-certs\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.595372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.595305    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-credential-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.595449 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.595415    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-data\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.595845 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.595823    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee52503f-b72e-42fe-a5b2-0a287941daca-istiod-ca-cert\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.597292 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.597269    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-envoy\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.597486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.597461    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-podinfo\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.616381 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.616347    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-token\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.616556 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.616474    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xbl\" (UniqueName: \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-kube-api-access-b9xbl\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:23.695698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:23.695606    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:24.030335 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:24.030300    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"]
Nov 02 14:33:24.032181 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:33:24.032145    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee52503f_b72e_42fe_a5b2_0a287941daca.slice/crio-31c80bd75954f00287f733ca359f691317a2ef7a0b0bd637e75d234dfdd97015 WatchSource:0}: Error finding container 31c80bd75954f00287f733ca359f691317a2ef7a0b0bd637e75d234dfdd97015: Status 404 returned error can't find the container with id 31c80bd75954f00287f733ca359f691317a2ef7a0b0bd637e75d234dfdd97015
Nov 02 14:33:24.035793 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:24.035759    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:33:24.035908 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:24.035831    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:33:24.035908 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:24.035876    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:33:24.646645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:24.646604    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" event={"ID":"ee52503f-b72e-42fe-a5b2-0a287941daca","Type":"ContainerStarted","Data":"34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96"}
Nov 02 14:33:24.647105 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:24.646685    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" event={"ID":"ee52503f-b72e-42fe-a5b2-0a287941daca","Type":"ContainerStarted","Data":"31c80bd75954f00287f733ca359f691317a2ef7a0b0bd637e75d234dfdd97015"}
Nov 02 14:33:24.689403 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:24.689333    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" podStartSLOduration=1.689313135 podStartE2EDuration="1.689313135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:24.687379379 +0000 UTC m=+2936.646754989" watchObservedRunningTime="2026-11-02 14:33:24.689313135 +0000 UTC m=+2936.648688746"
Nov 02 14:33:24.696045 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:24.696012    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:24.702551 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:24.702443    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"]
Nov 02 14:33:25.699038 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:25.698994    6284 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 14:33:25.699580 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:25.699067    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 14:33:26.696787 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:26.696750    6284 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 14:33:26.697045 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:26.696810    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 14:33:27.696320 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:27.696207    6284 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 14:33:27.696320 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:27.696283    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 14:33:28.697124 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:28.697077    6284 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 14:33:28.697604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:28.697159    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 14:33:29.696875 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:29.696838    6284 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 14:33:29.697124 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:29.696909    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 14:33:29.703115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:29.703077    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j" podUID="2e46e749-9ab2-48b1-b59e-872aef58c986" containerName="payload-processing" containerID="cri-o://2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d" gracePeriod=25
Nov 02 14:33:30.032661 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.032628    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:33:30.105902 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.105850    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-kube-api-access-992qd\" (UniqueName: \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-kube-api-access-992qd\") pod \"2e46e749-9ab2-48b1-b59e-872aef58c986\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") "
Nov 02 14:33:30.106158 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.106065    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-plugins-config-volume\") pod \"2e46e749-9ab2-48b1-b59e-872aef58c986\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") "
Nov 02 14:33:30.106245 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.106222    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-provider-credentials\") pod \"2e46e749-9ab2-48b1-b59e-872aef58c986\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") "
Nov 02 14:33:30.106408 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.106390    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-routing-overlay-volume\") pod \"2e46e749-9ab2-48b1-b59e-872aef58c986\" (UID: \"2e46e749-9ab2-48b1-b59e-872aef58c986\") "
Nov 02 14:33:30.106469 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.106452    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-plugins-config-volume" pod "2e46e749-9ab2-48b1-b59e-872aef58c986" (UID: "2e46e749-9ab2-48b1-b59e-872aef58c986"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:30.106986 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.106951    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-routing-overlay-volume" pod "2e46e749-9ab2-48b1-b59e-872aef58c986" (UID: "2e46e749-9ab2-48b1-b59e-872aef58c986"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:30.107148 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.107116    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:33:30.107148 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.107140    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e46e749-9ab2-48b1-b59e-872aef58c986-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:33:30.109056 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.109023    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-provider-credentials" pod "2e46e749-9ab2-48b1-b59e-872aef58c986" (UID: "2e46e749-9ab2-48b1-b59e-872aef58c986"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:30.109210 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.109185    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-kube-api-access-992qd" pod "2e46e749-9ab2-48b1-b59e-872aef58c986" (UID: "2e46e749-9ab2-48b1-b59e-872aef58c986"). InnerVolumeSpecName "kube-api-access-992qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:30.209309 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.209278    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:33:30.209309 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.209306    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-992qd\" (UniqueName: \"kubernetes.io/projected/2e46e749-9ab2-48b1-b59e-872aef58c986-kube-api-access-992qd\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:33:30.696706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.696651    6284 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 02 14:33:30.696919 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.696723    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 02 14:33:30.716468 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.716435    6284 generic.go:356] "Generic (PLEG): container finished" podID="2e46e749-9ab2-48b1-b59e-872aef58c986" containerID="2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d" exitCode=0
Nov 02 14:33:30.716985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.716494    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"
Nov 02 14:33:30.716985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.716491    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j" event={"ID":"2e46e749-9ab2-48b1-b59e-872aef58c986","Type":"ContainerDied","Data":"2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d"}
Nov 02 14:33:30.716985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.716617    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56f5855db-l7f7j" event={"ID":"2e46e749-9ab2-48b1-b59e-872aef58c986","Type":"ContainerDied","Data":"3044841693c94fe830871d195887f976e864317b15a6970c400b707d973d47aa"}
Nov 02 14:33:30.716985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.716641    6284 scope.go:122] "RemoveContainer" containerID="2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d"
Nov 02 14:33:30.727445 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.727426    6284 scope.go:122] "RemoveContainer" containerID="2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d"
Nov 02 14:33:30.727729 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:33:30.727708    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d\": container with ID starting with 2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d not found: ID does not exist" containerID="2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d"
Nov 02 14:33:30.727776 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.727739    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d"} err="failed to get container status \"2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d\": rpc error: code = NotFound desc = could not find container \"2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d\": container with ID starting with 2bdf69af4129d5545a23826b91d2715d216b3f122ffb81300fe5b03d02a7543d not found: ID does not exist"
Nov 02 14:33:30.777963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.777558    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"]
Nov 02 14:33:30.781222 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:30.781183    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56f5855db-l7f7j"]
Nov 02 14:33:31.701625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:31.701590    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:31.702068 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:31.702043    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:31.702827 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:31.702804    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:33:32.172101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:33:32.172068    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e46e749-9ab2-48b1-b59e-872aef58c986" path="/var/lib/kubelet/pods/2e46e749-9ab2-48b1-b59e-872aef58c986/volumes"
Nov 02 14:33:43.897552 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:33:43.897512    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:34:13.896678 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:34:13.896633    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:34:29.034187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:34:29.034148    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:34:29.037054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:34:29.037027    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:34:43.897083 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:34:43.897042    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:35:13.973994 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:35:13.973901    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:35:17.135921 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.135879    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"]
Nov 02 14:35:17.136603 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.136583    6284 cpu_manager.go:414] "containerMap: removing container" podUID="2e46e749-9ab2-48b1-b59e-872aef58c986" containerName="payload-processing"
Nov 02 14:35:17.136603 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.136603    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e46e749-9ab2-48b1-b59e-872aef58c986" containerName="payload-processing"
Nov 02 14:35:17.136760 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.136712    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e46e749-9ab2-48b1-b59e-872aef58c986" containerName="payload-processing"
Nov 02 14:35:17.141303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.141270    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.164205 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.164174    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-09d1a5-serving-cert\""
Nov 02 14:35:17.168975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.168919    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"]
Nov 02 14:35:17.245868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.245828    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-metrics-certs\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.246132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.245896    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5vcw\" (UniqueName: \"kubernetes.io/projected/106517f1-b322-462b-80ba-ae2c8316bc62-kube-api-access-c5vcw\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.246132 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.246099    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-maas-api-tls\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.246232 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.246148    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/106517f1-b322-462b-80ba-ae2c8316bc62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.350067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.349450    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-metrics-certs\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.357515 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.357465    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5vcw\" (UniqueName: \"kubernetes.io/projected/106517f1-b322-462b-80ba-ae2c8316bc62-kube-api-access-c5vcw\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.358576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.358537    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-maas-api-tls\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.360058 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.359698    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/106517f1-b322-462b-80ba-ae2c8316bc62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.362484 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.362000    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/106517f1-b322-462b-80ba-ae2c8316bc62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.396383 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.392605    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-maas-api-tls\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.396660 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.396623    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5vcw\" (UniqueName: \"kubernetes.io/projected/106517f1-b322-462b-80ba-ae2c8316bc62-kube-api-access-c5vcw\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.418038 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.418001    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-metrics-certs\") pod \"maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.457270 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.457225    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:17.770507 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.770475    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"]
Nov 02 14:35:17.772826 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:35:17.772791    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106517f1_b322_462b_80ba_ae2c8316bc62.slice/crio-a8c73607b239dadb4898026b4e7c9d169bcbc62e926f0552303c46a0adb5bd3f WatchSource:0}: Error finding container a8c73607b239dadb4898026b4e7c9d169bcbc62e926f0552303c46a0adb5bd3f: Status 404 returned error can't find the container with id a8c73607b239dadb4898026b4e7c9d169bcbc62e926f0552303c46a0adb5bd3f
Nov 02 14:35:17.774394 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.774371    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:35:17.944674 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:17.944630    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx" event={"ID":"106517f1-b322-462b-80ba-ae2c8316bc62","Type":"ContainerStarted","Data":"a8c73607b239dadb4898026b4e7c9d169bcbc62e926f0552303c46a0adb5bd3f"}
Nov 02 14:35:18.954065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:18.954029    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx" event={"ID":"106517f1-b322-462b-80ba-ae2c8316bc62","Type":"ContainerStarted","Data":"e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d"}
Nov 02 14:35:18.954487 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:18.954252    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:18.992295 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:18.992238    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx" podStartSLOduration=1.515822869 podStartE2EDuration="1.992221234s" totalImagesPullingTime="476.398365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:18.991432202 +0000 UTC m=+3050.950807808" watchObservedRunningTime="2026-11-02 14:35:18.992221234 +0000 UTC m=+3050.951596840"
Nov 02 14:35:19.155439 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:19.155397    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"]
Nov 02 14:35:24.156515 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.156427    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j" podUID="1cb8f4c6-be95-427e-b4a4-59e3d6b56e46" containerName="payload-processing" containerID="cri-o://9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab" gracePeriod=25
Nov 02 14:35:24.439751 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.439714    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:35:24.630098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.630057    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-kube-api-access-7xnsd\" (UniqueName: \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-kube-api-access-7xnsd\") pod \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") "
Nov 02 14:35:24.630290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.630124    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-plugins-config-volume\") pod \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") "
Nov 02 14:35:24.630290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.630275    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-routing-overlay-volume\") pod \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") "
Nov 02 14:35:24.630370 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.630329    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-provider-credentials\") pod \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\" (UID: \"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46\") "
Nov 02 14:35:24.630658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.630621    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-routing-overlay-volume" pod "1cb8f4c6-be95-427e-b4a4-59e3d6b56e46" (UID: "1cb8f4c6-be95-427e-b4a4-59e3d6b56e46"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:35:24.630787 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.630618    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-plugins-config-volume" pod "1cb8f4c6-be95-427e-b4a4-59e3d6b56e46" (UID: "1cb8f4c6-be95-427e-b4a4-59e3d6b56e46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:35:24.630787 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.630765    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:35:24.630787 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.630778    6284 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-routing-overlay-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:35:24.632476 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.632448    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-kube-api-access-7xnsd" pod "1cb8f4c6-be95-427e-b4a4-59e3d6b56e46" (UID: "1cb8f4c6-be95-427e-b4a4-59e3d6b56e46"). InnerVolumeSpecName "kube-api-access-7xnsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:35:24.632562 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.632485    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-provider-credentials" pod "1cb8f4c6-be95-427e-b4a4-59e3d6b56e46" (UID: "1cb8f4c6-be95-427e-b4a4-59e3d6b56e46"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:35:24.731918 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.731799    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xnsd\" (UniqueName: \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-kube-api-access-7xnsd\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:35:24.731918 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.731841    6284 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46-provider-credentials\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:35:24.967749 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:24.967715    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:35:25.005137 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:25.005099    6284 generic.go:356] "Generic (PLEG): container finished" podID="1cb8f4c6-be95-427e-b4a4-59e3d6b56e46" containerID="9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab" exitCode=0
Nov 02 14:35:25.005350 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:25.005218    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"
Nov 02 14:35:25.005405 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:25.005210    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j" event={"ID":"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46","Type":"ContainerDied","Data":"9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab"}
Nov 02 14:35:25.005445 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:25.005402    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j" event={"ID":"1cb8f4c6-be95-427e-b4a4-59e3d6b56e46","Type":"ContainerDied","Data":"8acd97eeed13273d4cd1ff2198ef7d12b42a943739e41c537b6ba143d097c868"}
Nov 02 14:35:25.005445 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:25.005434    6284 scope.go:122] "RemoveContainer" containerID="9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab"
Nov 02 14:35:25.018831 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:25.018799    6284 scope.go:122] "RemoveContainer" containerID="9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab"
Nov 02 14:35:25.019321 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:35:25.019288    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab\": container with ID starting with 9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab not found: ID does not exist" containerID="9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab"
Nov 02 14:35:25.019535 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:25.019332    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab"} err="failed to get container status \"9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab\": rpc error: code = NotFound desc = could not find container \"9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab\": container with ID starting with 9b8e455b778a60b1e671af5da6d0221d5fd570e461e51940ebb6b4c36bdc5aab not found: ID does not exist"
Nov 02 14:35:25.086486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:25.086445    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"]
Nov 02 14:35:25.097126 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:25.097081    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-26b508/payload-processing-external-model-e2e-extmod-a-26b508-8457r684j"]
Nov 02 14:35:26.182469 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:35:26.182426    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb8f4c6-be95-427e-b4a4-59e3d6b56e46" path="/var/lib/kubelet/pods/1cb8f4c6-be95-427e-b4a4-59e3d6b56e46/volumes"
Nov 02 14:35:43.896516 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:35:43.896473    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:36:05.167835 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.167758    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"]
Nov 02 14:36:05.169004 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.168982    6284 cpu_manager.go:414] "containerMap: removing container" podUID="1cb8f4c6-be95-427e-b4a4-59e3d6b56e46" containerName="payload-processing"
Nov 02 14:36:05.169004 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.169005    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb8f4c6-be95-427e-b4a4-59e3d6b56e46" containerName="payload-processing"
Nov 02 14:36:05.169143 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.169133    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb8f4c6-be95-427e-b4a4-59e3d6b56e46" containerName="payload-processing"
Nov 02 14:36:05.173107 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.173085    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.197851 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.197810    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e1fcb4-serving-cert\""
Nov 02 14:36:05.203361 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.203325    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"]
Nov 02 14:36:05.267061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.267020    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-metrics-certs\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.267335 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.267312    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lb2j\" (UniqueName: \"kubernetes.io/projected/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-kube-api-access-7lb2j\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.267443 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.267376    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.267527 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.267505    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.405010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.404192    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lb2j\" (UniqueName: \"kubernetes.io/projected/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-kube-api-access-7lb2j\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.405010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.404273    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.405010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.404317    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.405010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.404420    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-metrics-certs\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.407856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.407589    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.408439 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.408407    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-metrics-certs\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.416261 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.415989    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-maas-api-tls\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.438808 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.438727    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lb2j\" (UniqueName: \"kubernetes.io/projected/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-kube-api-access-7lb2j\") pod \"maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.487762 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.487726    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:05.800025 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:05.799994    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"]
Nov 02 14:36:05.802346 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:36:05.802304    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1a8118_ecdb_48a7_af76_2c1fe30f71ec.slice/crio-b6bfe891d8536387ed2cb7cc3f5e47d9f5c5d4fd9fcef5b9e44e1adacf32eff0 WatchSource:0}: Error finding container b6bfe891d8536387ed2cb7cc3f5e47d9f5c5d4fd9fcef5b9e44e1adacf32eff0: Status 404 returned error can't find the container with id b6bfe891d8536387ed2cb7cc3f5e47d9f5c5d4fd9fcef5b9e44e1adacf32eff0
Nov 02 14:36:06.345739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:06.345642    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9" event={"ID":"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec","Type":"ContainerStarted","Data":"cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed"}
Nov 02 14:36:06.345739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:06.345717    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9" event={"ID":"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec","Type":"ContainerStarted","Data":"b6bfe891d8536387ed2cb7cc3f5e47d9f5c5d4fd9fcef5b9e44e1adacf32eff0"}
Nov 02 14:36:06.346213 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:06.345748    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:06.428971 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:06.395724    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9" podStartSLOduration=1.120469908 podStartE2EDuration="1.395697898s" totalImagesPullingTime="275.22799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:06.395176029 +0000 UTC m=+3098.354551637" watchObservedRunningTime="2026-11-02 14:36:06.395697898 +0000 UTC m=+3098.355073707"
Nov 02 14:36:12.359722 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:36:12.359690    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:36:13.949659 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:36:13.949618    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:36:43.990518 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:36:43.990473    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:37:09.682754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.682707    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"]
Nov 02 14:37:09.688036 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.688010    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:09.709269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.709232    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d39fa6-serving-cert\""
Nov 02 14:37:09.725864 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.725719    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"]
Nov 02 14:37:09.866564 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.866522    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf597423-a322-4d66-a75f-6fdda939de05-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:09.866792 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.866641    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-metrics-certs\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:09.866901 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.866861    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5lk\" (UniqueName: \"kubernetes.io/projected/cf597423-a322-4d66-a75f-6fdda939de05-kube-api-access-cm5lk\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:09.867143 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.867122    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-maas-api-tls\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:09.972282 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.972178    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-maas-api-tls\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:09.972282 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.972241    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf597423-a322-4d66-a75f-6fdda939de05-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:09.972539 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.972308    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-metrics-certs\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:09.972539 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.972463    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5lk\" (UniqueName: \"kubernetes.io/projected/cf597423-a322-4d66-a75f-6fdda939de05-kube-api-access-cm5lk\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:09.999958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:09.974459    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf597423-a322-4d66-a75f-6fdda939de05-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:10.001191 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:10.001152    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-metrics-certs\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:10.013843 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:10.013806    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-maas-api-tls\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:10.066576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:10.066501    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5lk\" (UniqueName: \"kubernetes.io/projected/cf597423-a322-4d66-a75f-6fdda939de05-kube-api-access-cm5lk\") pod \"maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:10.331201 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:10.331146    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:10.751443 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:10.744488    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"]
Nov 02 14:37:10.755915 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:37:10.755873    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf597423_a322_4d66_a75f_6fdda939de05.slice/crio-d27f557d428971a3cc784da9783a96f66cbbc482f9dc2270fa660517a8f05ceb WatchSource:0}: Error finding container d27f557d428971a3cc784da9783a96f66cbbc482f9dc2270fa660517a8f05ceb: Status 404 returned error can't find the container with id d27f557d428971a3cc784da9783a96f66cbbc482f9dc2270fa660517a8f05ceb
Nov 02 14:37:10.990747 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:10.987656    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6" event={"ID":"cf597423-a322-4d66-a75f-6fdda939de05","Type":"ContainerStarted","Data":"d27f557d428971a3cc784da9783a96f66cbbc482f9dc2270fa660517a8f05ceb"}
Nov 02 14:37:11.996886 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:11.996840    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6" event={"ID":"cf597423-a322-4d66-a75f-6fdda939de05","Type":"ContainerStarted","Data":"691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225"}
Nov 02 14:37:11.997460 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:11.996983    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:12.040655 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:12.040597    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6" podStartSLOduration=2.5262449719999998 podStartE2EDuration="3.040578644s" totalImagesPullingTime="514.333672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:12.038089544 +0000 UTC m=+3163.997465152" watchObservedRunningTime="2026-11-02 14:37:12.040578644 +0000 UTC m=+3163.999954250"
Nov 02 14:37:13.895538 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:37:13.895491    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:37:18.009608 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:18.009575    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:37:31.688463 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.688429    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"]
Nov 02 14:37:31.693050 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.693023    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.720162 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.720129    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fe1881-serving-cert\""
Nov 02 14:37:31.725459 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.725398    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"]
Nov 02 14:37:31.855600 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.855567    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-metrics-certs\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.855600 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.855605    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-maas-api-tls\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.855808 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.855778    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq86l\" (UniqueName: \"kubernetes.io/projected/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-kube-api-access-cq86l\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.856005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.855985    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.960168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.958202    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-metrics-certs\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.960168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.958261    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-maas-api-tls\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.960168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.958427    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq86l\" (UniqueName: \"kubernetes.io/projected/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-kube-api-access-cq86l\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.960168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.958595    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.960168 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:37:31.959613    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-serving-cert: secret "maas-api-e2e-inf-a-fe1881-serving-cert" not found
Nov 02 14:37:31.960168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.959678    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.960168 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:37:31.960018    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-maas-api-tls podName:440dcea9-b109-41e6-aad5-bf7dcdaf4c7a nodeName:}" failed. No retries permitted until 2026-11-02 14:37:32.459991233 +0000 UTC m=+3184.419366825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-maas-api-tls") pod "maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v" (UID: "440dcea9-b109-41e6-aad5-bf7dcdaf4c7a") : secret "maas-api-e2e-inf-a-fe1881-serving-cert" not found
Nov 02 14:37:31.989427 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.989391    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-metrics-certs\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:31.991641 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:31.991607    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq86l\" (UniqueName: \"kubernetes.io/projected/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-kube-api-access-cq86l\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:32.467329 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:32.467290    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-maas-api-tls\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:32.470094 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:32.470055    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-maas-api-tls\") pod \"maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:32.608966 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:32.608207    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:33.049044 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:33.048459    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"]
Nov 02 14:37:33.228309 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:33.228261    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v" event={"ID":"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a","Type":"ContainerStarted","Data":"2dd71069e8ed05d62a5e40756aef4bfe19a8cd6f2ca5e71397ec9ad14c81d2f2"}
Nov 02 14:37:34.237327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:34.237291    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v" event={"ID":"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a","Type":"ContainerStarted","Data":"e6c6f9cb63bb85fbac76f307c177abf993ce122e43f15e78fd3db36190bb508c"}
Nov 02 14:37:34.237773 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:34.237419    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:34.280460 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:34.280408    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v" podStartSLOduration=2.798351067 podStartE2EDuration="3.280393102s" totalImagesPullingTime="482.042035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:34.275665097 +0000 UTC m=+3186.235040704" watchObservedRunningTime="2026-11-02 14:37:34.280393102 +0000 UTC m=+3186.239768708"
Nov 02 14:37:40.249510 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:37:40.249474    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:37:43.897262 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:37:43.897218    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:38:01.251120 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.251081    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"]
Nov 02 14:38:01.255746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.255722    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.280092 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.280058    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e1136e-serving-cert\""
Nov 02 14:38:01.285063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.285027    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"]
Nov 02 14:38:01.372143 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.372104    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-metrics-certs\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.372359 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.372275    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnq4c\" (UniqueName: \"kubernetes.io/projected/0059df5c-839a-49af-8f54-2316292dbc49-kube-api-access-nnq4c\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.372442 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.372376    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0059df5c-839a-49af-8f54-2316292dbc49-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.372526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.372508    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-maas-api-tls\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.487796 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.487757    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-metrics-certs\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.488197 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.488168    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnq4c\" (UniqueName: \"kubernetes.io/projected/0059df5c-839a-49af-8f54-2316292dbc49-kube-api-access-nnq4c\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.488348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.488255    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0059df5c-839a-49af-8f54-2316292dbc49-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.488433 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.488375    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-maas-api-tls\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.489807 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.489780    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0059df5c-839a-49af-8f54-2316292dbc49-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.492861 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.492419    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-metrics-certs\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.493502 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.493472    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-maas-api-tls\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.510371 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.510292    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnq4c\" (UniqueName: \"kubernetes.io/projected/0059df5c-839a-49af-8f54-2316292dbc49-kube-api-access-nnq4c\") pod \"maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.571296 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.571244    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:01.891362 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:01.891325    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"]
Nov 02 14:38:01.893766 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:38:01.893738    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0059df5c_839a_49af_8f54_2316292dbc49.slice/crio-ebbb1174e86470a1f60e5765f072d4079a12551991fe07475c249d9d4aafdf2d WatchSource:0}: Error finding container ebbb1174e86470a1f60e5765f072d4079a12551991fe07475c249d9d4aafdf2d: Status 404 returned error can't find the container with id ebbb1174e86470a1f60e5765f072d4079a12551991fe07475c249d9d4aafdf2d
Nov 02 14:38:02.508907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:02.508875    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv" event={"ID":"0059df5c-839a-49af-8f54-2316292dbc49","Type":"ContainerStarted","Data":"8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a"}
Nov 02 14:38:02.509576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:02.508928    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv" event={"ID":"0059df5c-839a-49af-8f54-2316292dbc49","Type":"ContainerStarted","Data":"ebbb1174e86470a1f60e5765f072d4079a12551991fe07475c249d9d4aafdf2d"}
Nov 02 14:38:02.509576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:02.509036    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:02.556188 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:02.556110    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv" podStartSLOduration=1.240005215 podStartE2EDuration="1.556084767s" totalImagesPullingTime="316.079552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:02.546351826 +0000 UTC m=+3214.505727434" watchObservedRunningTime="2026-11-02 14:38:02.556084767 +0000 UTC m=+3214.515460376"
Nov 02 14:38:08.563741 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:08.563711    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:38:11.824280 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.822654    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"]
Nov 02 14:38:11.830516 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.830469    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:11.847160 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.847125    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"]
Nov 02 14:38:11.852110 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.852080    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-09d1a5\"/\"openshift-service-ca.crt\""
Nov 02 14:38:11.852110 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.852082    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-09d1a5\"/\"kube-root-ca.crt\""
Nov 02 14:38:11.858154 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.858122    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-09d1a5\"/\"test-model-09d1a5-kserve-self-signed-certs\""
Nov 02 14:38:11.858446 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.858403    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-09d1a5\"/\"default-dockercfg-2gw5k\""
Nov 02 14:38:11.908982 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.908911    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-home\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:11.909203 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.909034    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-model-cache\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:11.909203 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.909099    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8t2z\" (UniqueName: \"kubernetes.io/projected/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-kube-api-access-x8t2z\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:11.909203 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.909139    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tmp-dir\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:11.909327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.909201    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-dshm\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:11.909327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:11.909307    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tls-certs\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.010625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.010576    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-home\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.010625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.010629    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-model-cache\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.010904 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.010658    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8t2z\" (UniqueName: \"kubernetes.io/projected/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-kube-api-access-x8t2z\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.010904 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.010686    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tmp-dir\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.010904 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.010722    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-dshm\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.010904 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.010821    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tls-certs\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.011142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.011097    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-home\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.011435 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.011411    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-model-cache\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.011532 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.011507    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tmp-dir\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.014098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.014071    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-dshm\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.014266 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.014253    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tls-certs\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.032187 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.032143    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8t2z\" (UniqueName: \"kubernetes.io/projected/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-kube-api-access-x8t2z\") pod \"test-model-09d1a5-kserve-7cb7bf6b8c-msfkd\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") " pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.160282 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.160184    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:12.567753 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.567709    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"]
Nov 02 14:38:12.572839 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:38:12.572804    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5bbcbf6_e1fa_40c5_a873_8f5e2e27beb9.slice/crio-7dad5cffacdc5123f35b73fad0054ebb8a521c61b7aaf782f98b85fbcb27ddcc WatchSource:0}: Error finding container 7dad5cffacdc5123f35b73fad0054ebb8a521c61b7aaf782f98b85fbcb27ddcc: Status 404 returned error can't find the container with id 7dad5cffacdc5123f35b73fad0054ebb8a521c61b7aaf782f98b85fbcb27ddcc
Nov 02 14:38:12.608081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:12.608042    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd" event={"ID":"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9","Type":"ContainerStarted","Data":"7dad5cffacdc5123f35b73fad0054ebb8a521c61b7aaf782f98b85fbcb27ddcc"}
Nov 02 14:38:13.618299 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:13.618256    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd" event={"ID":"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9","Type":"ContainerStarted","Data":"cdc0312627645a11ebfb71c4afbae863f298ac8ce5156acc86ddb716d31b22c0"}
Nov 02 14:38:13.659517 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:13.659453    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd" podStartSLOduration=2.659432878 podStartE2EDuration="2.659432878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:13.655256655 +0000 UTC m=+3225.614632261" watchObservedRunningTime="2026-11-02 14:38:13.659432878 +0000 UTC m=+3225.618808484"
Nov 02 14:38:13.896672 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:38:13.896585    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:38:14.171000 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:14.170913    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:18.336031 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:18.335989    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"]
Nov 02 14:38:18.336496 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:18.336193    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp" podUID="cb5658b0-11e9-402f-8685-1419b07b20a8" containerName="sim" containerID="cri-o://a632d16757520a07f6329ea21e911b8358a31772d1ebd075ace2bc786be9ca84" gracePeriod=30
Nov 02 14:38:18.684892 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:18.684778    6284 generic.go:356] "Generic (PLEG): container finished" podID="cb5658b0-11e9-402f-8685-1419b07b20a8" containerID="a632d16757520a07f6329ea21e911b8358a31772d1ebd075ace2bc786be9ca84" exitCode=0
Nov 02 14:38:18.685228 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:18.685123    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp" event={"ID":"cb5658b0-11e9-402f-8685-1419b07b20a8","Type":"ContainerDied","Data":"a632d16757520a07f6329ea21e911b8358a31772d1ebd075ace2bc786be9ca84"}
Nov 02 14:38:18.877640 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:18.877610    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:38:18.903495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:18.903453    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb5658b0-11e9-402f-8685-1419b07b20a8-tls\" (UniqueName: \"kubernetes.io/secret/cb5658b0-11e9-402f-8685-1419b07b20a8-tls\") pod \"cb5658b0-11e9-402f-8685-1419b07b20a8\" (UID: \"cb5658b0-11e9-402f-8685-1419b07b20a8\") "
Nov 02 14:38:18.903689 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:18.903543    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5658b0-11e9-402f-8685-1419b07b20a8-kube-api-access-4hz9n\" (UniqueName: \"kubernetes.io/projected/cb5658b0-11e9-402f-8685-1419b07b20a8-kube-api-access-4hz9n\") pod \"cb5658b0-11e9-402f-8685-1419b07b20a8\" (UID: \"cb5658b0-11e9-402f-8685-1419b07b20a8\") "
Nov 02 14:38:18.905800 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:18.905767    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5658b0-11e9-402f-8685-1419b07b20a8-tls" pod "cb5658b0-11e9-402f-8685-1419b07b20a8" (UID: "cb5658b0-11e9-402f-8685-1419b07b20a8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:38:18.905928 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:18.905810    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5658b0-11e9-402f-8685-1419b07b20a8-kube-api-access-4hz9n" pod "cb5658b0-11e9-402f-8685-1419b07b20a8" (UID: "cb5658b0-11e9-402f-8685-1419b07b20a8"). InnerVolumeSpecName "kube-api-access-4hz9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:38:19.018010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:19.012168    6284 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/cb5658b0-11e9-402f-8685-1419b07b20a8-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:38:19.018010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:19.012213    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hz9n\" (UniqueName: \"kubernetes.io/projected/cb5658b0-11e9-402f-8685-1419b07b20a8-kube-api-access-4hz9n\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:38:19.694013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:19.693978    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"
Nov 02 14:38:19.694013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:19.693995    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp" event={"ID":"cb5658b0-11e9-402f-8685-1419b07b20a8","Type":"ContainerDied","Data":"ba15c10dce7de22e48897c941e8d08b7263141a4507448569be467ce722b2541"}
Nov 02 14:38:19.694561 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:19.694049    6284 scope.go:122] "RemoveContainer" containerID="a632d16757520a07f6329ea21e911b8358a31772d1ebd075ace2bc786be9ca84"
Nov 02 14:38:19.748258 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:19.748216    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"]
Nov 02 14:38:19.751923 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:19.751887    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-7bfb1f/e2e-usage-sim-99ce8c-769cb9559f-87qmp"]
Nov 02 14:38:20.182320 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:20.182288    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5658b0-11e9-402f-8685-1419b07b20a8" path="/var/lib/kubelet/pods/cb5658b0-11e9-402f-8685-1419b07b20a8/volumes"
Nov 02 14:38:22.171715 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:22.171630    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:22.176600 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:22.176574    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:22.731946 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:22.731910    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:38:34.284690 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.284655    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"]
Nov 02 14:38:34.286179 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.286150    6284 cpu_manager.go:414] "containerMap: removing container" podUID="cb5658b0-11e9-402f-8685-1419b07b20a8" containerName="sim"
Nov 02 14:38:34.286179 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.286177    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5658b0-11e9-402f-8685-1419b07b20a8" containerName="sim"
Nov 02 14:38:34.286347 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.286316    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5658b0-11e9-402f-8685-1419b07b20a8" containerName="sim"
Nov 02 14:38:34.292024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.291995    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.314967 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.314914    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d39fa6\"/\"openshift-service-ca.crt\""
Nov 02 14:38:34.315195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.314953    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d39fa6\"/\"default-dockercfg-5pgh5\""
Nov 02 14:38:34.317030 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.316164    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d39fa6\"/\"kube-root-ca.crt\""
Nov 02 14:38:34.317030 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.316363    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d39fa6\"/\"test-model-d39fa6-kserve-self-signed-certs\""
Nov 02 14:38:34.320981 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.320923    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"]
Nov 02 14:38:34.415923 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.415880    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxrg6\" (UniqueName: \"kubernetes.io/projected/05adda46-8560-440c-a0f6-8ae9252bb27a-kube-api-access-xxrg6\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.415923 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.415948    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-tmp-dir\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.416239 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.416150    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-home\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.416308 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.416290    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05adda46-8560-440c-a0f6-8ae9252bb27a-tls-certs\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.416493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.416461    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-model-cache\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.416550 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.416534    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-dshm\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.518176 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.518139    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05adda46-8560-440c-a0f6-8ae9252bb27a-tls-certs\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.518390 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.518269    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-model-cache\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.518390 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.518314    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-dshm\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.518523 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.518426    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxrg6\" (UniqueName: \"kubernetes.io/projected/05adda46-8560-440c-a0f6-8ae9252bb27a-kube-api-access-xxrg6\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.518523 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.518476    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-tmp-dir\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.518632 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.518584    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-home\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.519120 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.519089    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-home\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.519440 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.519414    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-model-cache\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.519593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.519447    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-tmp-dir\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.522196 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.522165    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05adda46-8560-440c-a0f6-8ae9252bb27a-tls-certs\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.535993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.535889    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-dshm\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.545572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.545531    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxrg6\" (UniqueName: \"kubernetes.io/projected/05adda46-8560-440c-a0f6-8ae9252bb27a-kube-api-access-xxrg6\") pod \"test-model-d39fa6-kserve-7b6dc848df-skf22\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") " pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.609185 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:34.609149    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:34.987952 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:38:34.987859    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05adda46_8560_440c_a0f6_8ae9252bb27a.slice/crio-3c36a63161ba8746d1a2c2c1d1dce04909e764529d6542775b88aea5125ae11e WatchSource:0}: Error finding container 3c36a63161ba8746d1a2c2c1d1dce04909e764529d6542775b88aea5125ae11e: Status 404 returned error can't find the container with id 3c36a63161ba8746d1a2c2c1d1dce04909e764529d6542775b88aea5125ae11e
Nov 02 14:38:35.008608 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:35.008578    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"]
Nov 02 14:38:35.934738 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:35.934680    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22" event={"ID":"05adda46-8560-440c-a0f6-8ae9252bb27a","Type":"ContainerStarted","Data":"1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79"}
Nov 02 14:38:35.934738 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:35.934745    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22" event={"ID":"05adda46-8560-440c-a0f6-8ae9252bb27a","Type":"ContainerStarted","Data":"3c36a63161ba8746d1a2c2c1d1dce04909e764529d6542775b88aea5125ae11e"}
Nov 02 14:38:35.977621 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:35.977559    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22" podStartSLOduration=1.9775444549999999 podStartE2EDuration="1.977544455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:35.975085475 +0000 UTC m=+3247.934461082" watchObservedRunningTime="2026-11-02 14:38:35.977544455 +0000 UTC m=+3247.936920062"
Nov 02 14:38:36.609905 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:36.609867    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:43.490630 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.490587    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"]
Nov 02 14:38:43.548621 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.548571    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.558250 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.558210    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"]
Nov 02 14:38:43.576441 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.572048    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-16c822-serving-cert\""
Nov 02 14:38:43.748428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.748325    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-metrics-certs\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.748624 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.748547    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-maas-api-tls\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.748624 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.748593    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrv59\" (UniqueName: \"kubernetes.io/projected/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-kube-api-access-nrv59\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.748826 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.748796    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.850839 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.850795    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-metrics-certs\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.851061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.850907    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-maas-api-tls\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.851061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.850947    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrv59\" (UniqueName: \"kubernetes.io/projected/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-kube-api-access-nrv59\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.851061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.851020    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.851061 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:38:43.851046    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-serving-cert: secret "maas-api-e2e-inf-a-16c822-serving-cert" not found
Nov 02 14:38:43.851342 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:38:43.851124    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-maas-api-tls podName:57fdfa91-1d6e-4cf5-876d-c586468c1e0e nodeName:}" failed. No retries permitted until 2026-11-02 14:38:44.351101205 +0000 UTC m=+3256.310476808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-maas-api-tls") pod "maas-api-e2e-inf-a-16c822-d6f97496d-mclsg" (UID: "57fdfa91-1d6e-4cf5-876d-c586468c1e0e") : secret "maas-api-e2e-inf-a-16c822-serving-cert" not found
Nov 02 14:38:43.851747 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.851725    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.853463 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.853433    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-metrics-certs\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.880742 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:43.880703    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrv59\" (UniqueName: \"kubernetes.io/projected/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-kube-api-access-nrv59\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:43.898763 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:38:43.898724    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:38:44.358634 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:44.358592    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-maas-api-tls\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:44.361331 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:44.361300    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-maas-api-tls\") pod \"maas-api-e2e-inf-a-16c822-d6f97496d-mclsg\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:44.489164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:44.489110    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:44.663010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:44.622370    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:44.691205 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:44.689607    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:44.806075 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:44.806045    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"]
Nov 02 14:38:44.807864 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:38:44.807838    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fdfa91_1d6e_4cf5_876d_c586468c1e0e.slice/crio-21618be937c94b24d1fbe670c97dd5f69ff896eb9ce1df608a399ed6c97add25 WatchSource:0}: Error finding container 21618be937c94b24d1fbe670c97dd5f69ff896eb9ce1df608a399ed6c97add25: Status 404 returned error can't find the container with id 21618be937c94b24d1fbe670c97dd5f69ff896eb9ce1df608a399ed6c97add25
Nov 02 14:38:45.085559 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:45.085517    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg" event={"ID":"57fdfa91-1d6e-4cf5-876d-c586468c1e0e","Type":"ContainerStarted","Data":"21618be937c94b24d1fbe670c97dd5f69ff896eb9ce1df608a399ed6c97add25"}
Nov 02 14:38:45.098097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:45.098061    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:38:46.095490 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:46.095443    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg" event={"ID":"57fdfa91-1d6e-4cf5-876d-c586468c1e0e","Type":"ContainerStarted","Data":"a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674"}
Nov 02 14:38:46.096054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:46.095568    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:38:46.143162 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:46.143091    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg" podStartSLOduration=2.713015783 podStartE2EDuration="3.143072494s" totalImagesPullingTime="430.056711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:46.139036334 +0000 UTC m=+3258.098411943" watchObservedRunningTime="2026-11-02 14:38:46.143072494 +0000 UTC m=+3258.102448171"
Nov 02 14:38:52.113727 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:38:52.113696    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:39:09.916861 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:09.916819    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"]
Nov 02 14:39:09.924690 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:09.924657    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:09.944561 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:09.944525    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"]
Nov 02 14:39:09.949871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:09.949839    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e1fcb4\"/\"default-dockercfg-5jjwm\""
Nov 02 14:39:09.950111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:09.949959    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e1fcb4\"/\"test-model-e1fcb4-kserve-self-signed-certs\""
Nov 02 14:39:10.006139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.006101    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e1fcb4\"/\"openshift-service-ca.crt\""
Nov 02 14:39:10.006431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.006410    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e1fcb4\"/\"kube-root-ca.crt\""
Nov 02 14:39:10.048444 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.048400    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbbl\" (UniqueName: \"kubernetes.io/projected/ad4dea4c-9cda-4aae-88be-a31096fcb38d-kube-api-access-cfbbl\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.048661 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.048468    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-model-cache\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.048661 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.048565    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tmp-dir\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.048826 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.048806    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-dshm\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.048873 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.048864    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tls-certs\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.049134 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.049105    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-home\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.152168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.152128    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-dshm\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.152395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.152213    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tls-certs\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.152395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.152269    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-home\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.152395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.152308    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbbl\" (UniqueName: \"kubernetes.io/projected/ad4dea4c-9cda-4aae-88be-a31096fcb38d-kube-api-access-cfbbl\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.152395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.152365    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-model-cache\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.152612 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.152391    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tmp-dir\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.161969 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.153016    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tmp-dir\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.165962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.163373    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-home\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.178288 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.174727    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-model-cache\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.189958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.179837    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-dshm\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.225452 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.225413    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tls-certs\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.241962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.240699    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbbl\" (UniqueName: \"kubernetes.io/projected/ad4dea4c-9cda-4aae-88be-a31096fcb38d-kube-api-access-cfbbl\") pod \"test-model-e1fcb4-kserve-8594fc84df-x5zd9\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") " pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.540441 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.540401    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:10.867992 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:10.867961    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"]
Nov 02 14:39:10.870267 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:39:10.870235    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad4dea4c_9cda_4aae_88be_a31096fcb38d.slice/crio-6c09281765413399ce758621da053524ef7a242e535f3c8526a20d0fea705389 WatchSource:0}: Error finding container 6c09281765413399ce758621da053524ef7a242e535f3c8526a20d0fea705389: Status 404 returned error can't find the container with id 6c09281765413399ce758621da053524ef7a242e535f3c8526a20d0fea705389
Nov 02 14:39:11.316843 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:11.316805    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9" event={"ID":"ad4dea4c-9cda-4aae-88be-a31096fcb38d","Type":"ContainerStarted","Data":"32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e"}
Nov 02 14:39:11.317292 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:11.316854    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9" event={"ID":"ad4dea4c-9cda-4aae-88be-a31096fcb38d","Type":"ContainerStarted","Data":"6c09281765413399ce758621da053524ef7a242e535f3c8526a20d0fea705389"}
Nov 02 14:39:11.356819 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:11.356758    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9" podStartSLOduration=2.356740612 podStartE2EDuration="2.356740612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:11.352898384 +0000 UTC m=+3283.312273991" watchObservedRunningTime="2026-11-02 14:39:11.356740612 +0000 UTC m=+3283.316116216"
Nov 02 14:39:11.554521 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:11.554469    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:13.897558 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:39:13.897511    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:39:20.540735 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:20.540695    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:20.553737 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:20.553696    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:21.410902 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:21.410865    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:39:29.091812 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:29.091777    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:39:29.097518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:29.097484    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:39:32.509762 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.509726    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"]
Nov 02 14:39:32.515827 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.515798    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.536767 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.536728    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"]
Nov 02 14:39:32.541192 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.541152    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c8f647-serving-cert\""
Nov 02 14:39:32.578434 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.578386    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.578687 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.578623    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de777481-cbf8-4999-a4c6-d1b2687d2afe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.578687 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.578669    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj24k\" (UniqueName: \"kubernetes.io/projected/de777481-cbf8-4999-a4c6-d1b2687d2afe-kube-api-access-fj24k\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.578822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.578798    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-metrics-certs\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.680698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.680654    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-metrics-certs\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.680895 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.680832    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.680977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.680916    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de777481-cbf8-4999-a4c6-d1b2687d2afe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.680977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.680950    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj24k\" (UniqueName: \"kubernetes.io/projected/de777481-cbf8-4999-a4c6-d1b2687d2afe-kube-api-access-fj24k\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.681082 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:39:32.681018    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-serving-cert: secret "maas-api-e2e-inf-a-c8f647-serving-cert" not found
Nov 02 14:39:32.681130 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:39:32.681104    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-maas-api-tls podName:de777481-cbf8-4999-a4c6-d1b2687d2afe nodeName:}" failed. No retries permitted until 2026-11-02 14:39:33.181083037 +0000 UTC m=+3305.140458622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-maas-api-tls") pod "maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5" (UID: "de777481-cbf8-4999-a4c6-d1b2687d2afe") : secret "maas-api-e2e-inf-a-c8f647-serving-cert" not found
Nov 02 14:39:32.681796 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.681774    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de777481-cbf8-4999-a4c6-d1b2687d2afe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.683416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.683384    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-metrics-certs\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:32.733512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:32.732681    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj24k\" (UniqueName: \"kubernetes.io/projected/de777481-cbf8-4999-a4c6-d1b2687d2afe-kube-api-access-fj24k\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:33.189228 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:33.189178    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:33.192116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:33.192071    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:33.431872 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:33.431833    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:33.848296 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:33.848258    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"]
Nov 02 14:39:33.855403 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:39:33.855366    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde777481_cbf8_4999_a4c6_d1b2687d2afe.slice/crio-3a5eec5f334c8ab9f481191bd7e576371bd16cd03e67c3ad9b0954fac085e96f WatchSource:0}: Error finding container 3a5eec5f334c8ab9f481191bd7e576371bd16cd03e67c3ad9b0954fac085e96f: Status 404 returned error can't find the container with id 3a5eec5f334c8ab9f481191bd7e576371bd16cd03e67c3ad9b0954fac085e96f
Nov 02 14:39:34.521727 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:34.521681    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5" event={"ID":"de777481-cbf8-4999-a4c6-d1b2687d2afe","Type":"ContainerStarted","Data":"cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d"}
Nov 02 14:39:34.522021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:34.521758    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5" event={"ID":"de777481-cbf8-4999-a4c6-d1b2687d2afe","Type":"ContainerStarted","Data":"3a5eec5f334c8ab9f481191bd7e576371bd16cd03e67c3ad9b0954fac085e96f"}
Nov 02 14:39:34.522021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:34.521806    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:34.610900 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:34.610825    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5" podStartSLOduration=2.309030539 podStartE2EDuration="2.610803563s" totalImagesPullingTime="301.773024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:34.605258978 +0000 UTC m=+3306.564634585" watchObservedRunningTime="2026-11-02 14:39:34.610803563 +0000 UTC m=+3306.570179171"
Nov 02 14:39:39.652050 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.652005    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"]
Nov 02 14:39:39.657563 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.657528    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.680455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.680420    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cbde78-serving-cert\""
Nov 02 14:39:39.686992 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.686460    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"]
Nov 02 14:39:39.761683 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.761645    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-metrics-certs\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.761899 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.761762    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-maas-api-tls\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.761969 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.761920    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e020ed5-2443-4833-b519-d6ddeb069b93-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.762053 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.762037    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w9mc\" (UniqueName: \"kubernetes.io/projected/6e020ed5-2443-4833-b519-d6ddeb069b93-kube-api-access-6w9mc\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.863296 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.863249    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e020ed5-2443-4833-b519-d6ddeb069b93-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.863507 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.863340    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w9mc\" (UniqueName: \"kubernetes.io/projected/6e020ed5-2443-4833-b519-d6ddeb069b93-kube-api-access-6w9mc\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.863577 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.863512    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-metrics-certs\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.863638 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.863578    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-maas-api-tls\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.864088 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.864060    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e020ed5-2443-4833-b519-d6ddeb069b93-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.866336 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.866307    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-metrics-certs\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.866476 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.866449    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-maas-api-tls\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.885143 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.885108    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w9mc\" (UniqueName: \"kubernetes.io/projected/6e020ed5-2443-4833-b519-d6ddeb069b93-kube-api-access-6w9mc\") pod \"maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:39.974146 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:39.973762    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:40.300182 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:40.300155    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"]
Nov 02 14:39:40.302752 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:39:40.302718    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e020ed5_2443_4833_b519_d6ddeb069b93.slice/crio-efaaab0e7dcf0b3f1da603dbe14412f9cc5b740c2863d2a46e9dab944ace7f80 WatchSource:0}: Error finding container efaaab0e7dcf0b3f1da603dbe14412f9cc5b740c2863d2a46e9dab944ace7f80: Status 404 returned error can't find the container with id efaaab0e7dcf0b3f1da603dbe14412f9cc5b740c2863d2a46e9dab944ace7f80
Nov 02 14:39:40.566031 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:40.551693    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:39:40.577557 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:40.577517    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5" event={"ID":"6e020ed5-2443-4833-b519-d6ddeb069b93","Type":"ContainerStarted","Data":"efaaab0e7dcf0b3f1da603dbe14412f9cc5b740c2863d2a46e9dab944ace7f80"}
Nov 02 14:39:41.589238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:41.589187    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5" event={"ID":"6e020ed5-2443-4833-b519-d6ddeb069b93","Type":"ContainerStarted","Data":"22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a"}
Nov 02 14:39:41.589746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:41.589279    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:39:41.630754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:41.630644    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5" podStartSLOduration=2.22246285 podStartE2EDuration="2.630626319s" totalImagesPullingTime="408.163469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:41.628786212 +0000 UTC m=+3313.588161821" watchObservedRunningTime="2026-11-02 14:39:41.630626319 +0000 UTC m=+3313.590001926"
Nov 02 14:39:43.897589 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:39:43.897553    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:39:47.604250 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:39:47.604217    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:40:13.900475 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:40:13.900426    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:40:24.952801 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:24.952752    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"]
Nov 02 14:40:24.953486 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:24.952979    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerName="istio-proxy" containerID="cri-o://34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96" gracePeriod=30
Nov 02 14:40:29.937893 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:29.937857    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"]
Nov 02 14:40:29.938338 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:29.938105    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerName="istio-proxy" containerID="cri-o://256e06abd4347c3c130af60ec73300901e8c9c7f5903cdde0dfedfeba3448d02" gracePeriod=30
Nov 02 14:40:30.261833 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.261804    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:40:30.417504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.417457    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-kube-api-access-b9xbl\" (UniqueName: \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-kube-api-access-b9xbl\") pod \"ee52503f-b72e-42fe-a5b2-0a287941daca\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") "
Nov 02 14:40:30.419824 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.419796    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-credential-socket\") pod \"ee52503f-b72e-42fe-a5b2-0a287941daca\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") "
Nov 02 14:40:30.421019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.420950    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-credential-socket" pod "ee52503f-b72e-42fe-a5b2-0a287941daca" (UID: "ee52503f-b72e-42fe-a5b2-0a287941daca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:30.421216 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.421056    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-socket\") pod \"ee52503f-b72e-42fe-a5b2-0a287941daca\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") "
Nov 02 14:40:30.421216 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.421133    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-data\") pod \"ee52503f-b72e-42fe-a5b2-0a287941daca\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") "
Nov 02 14:40:30.421216 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.421163    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-socket" pod "ee52503f-b72e-42fe-a5b2-0a287941daca" (UID: "ee52503f-b72e-42fe-a5b2-0a287941daca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:30.421438 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.421329    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee52503f-b72e-42fe-a5b2-0a287941daca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee52503f-b72e-42fe-a5b2-0a287941daca-istiod-ca-cert\") pod \"ee52503f-b72e-42fe-a5b2-0a287941daca\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") "
Nov 02 14:40:30.421438 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.421402    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-certs\") pod \"ee52503f-b72e-42fe-a5b2-0a287941daca\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") "
Nov 02 14:40:30.421555 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.421474    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-envoy\") pod \"ee52503f-b72e-42fe-a5b2-0a287941daca\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") "
Nov 02 14:40:30.421555 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.421491    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-data" pod "ee52503f-b72e-42fe-a5b2-0a287941daca" (UID: "ee52503f-b72e-42fe-a5b2-0a287941daca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:30.421672 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.421554    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-podinfo\") pod \"ee52503f-b72e-42fe-a5b2-0a287941daca\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") "
Nov 02 14:40:30.421672 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.421588    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-token\" (UniqueName: \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-token\") pod \"ee52503f-b72e-42fe-a5b2-0a287941daca\" (UID: \"ee52503f-b72e-42fe-a5b2-0a287941daca\") "
Nov 02 14:40:30.421834 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.421806    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee52503f-b72e-42fe-a5b2-0a287941daca-istiod-ca-cert" pod "ee52503f-b72e-42fe-a5b2-0a287941daca" (UID: "ee52503f-b72e-42fe-a5b2-0a287941daca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:40:30.422112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.422085    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-certs" pod "ee52503f-b72e-42fe-a5b2-0a287941daca" (UID: "ee52503f-b72e-42fe-a5b2-0a287941daca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:30.422658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.422565    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:30.422658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.422593    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:30.422658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.422608    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:30.422658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.422622    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee52503f-b72e-42fe-a5b2-0a287941daca-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:30.422658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.422635    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:30.425135 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.425107    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-token" pod "ee52503f-b72e-42fe-a5b2-0a287941daca" (UID: "ee52503f-b72e-42fe-a5b2-0a287941daca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:30.425253 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.425145    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-kube-api-access-b9xbl" pod "ee52503f-b72e-42fe-a5b2-0a287941daca" (UID: "ee52503f-b72e-42fe-a5b2-0a287941daca"). InnerVolumeSpecName "kube-api-access-b9xbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:30.425253 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.425180    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-podinfo" pod "ee52503f-b72e-42fe-a5b2-0a287941daca" (UID: "ee52503f-b72e-42fe-a5b2-0a287941daca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:40:30.425764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.425735    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-envoy" pod "ee52503f-b72e-42fe-a5b2-0a287941daca" (UID: "ee52503f-b72e-42fe-a5b2-0a287941daca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:30.531688 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.523741    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xbl\" (UniqueName: \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-kube-api-access-b9xbl\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:30.531688 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.523786    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:30.531688 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.523800    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:30.531688 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:30.523813    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee52503f-b72e-42fe-a5b2-0a287941daca-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:31.050171 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:31.050130    6284 generic.go:356] "Generic (PLEG): container finished" podID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerID="34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96" exitCode=0
Nov 02 14:40:31.050630 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:31.050186    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"
Nov 02 14:40:31.050630 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:31.050229    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" event={"ID":"ee52503f-b72e-42fe-a5b2-0a287941daca","Type":"ContainerDied","Data":"34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96"}
Nov 02 14:40:31.050630 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:31.050296    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6" event={"ID":"ee52503f-b72e-42fe-a5b2-0a287941daca","Type":"ContainerDied","Data":"31c80bd75954f00287f733ca359f691317a2ef7a0b0bd637e75d234dfdd97015"}
Nov 02 14:40:31.050630 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:31.050325    6284 scope.go:122] "RemoveContainer" containerID="34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96"
Nov 02 14:40:31.062812 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:31.062790    6284 scope.go:122] "RemoveContainer" containerID="34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96"
Nov 02 14:40:31.063221 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:40:31.063196    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96\": container with ID starting with 34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96 not found: ID does not exist" containerID="34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96"
Nov 02 14:40:31.063323 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:31.063229    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96"} err="failed to get container status \"34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96\": rpc error: code = NotFound desc = could not find container \"34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96\": container with ID starting with 34d4c7223e3b9b1f0eab00ef43c999abc918fe3295fe383bf2abcf08a5b94f96 not found: ID does not exist"
Nov 02 14:40:31.099484 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:31.099447    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"]
Nov 02 14:40:31.110138 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:31.110104    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-z79h6"]
Nov 02 14:40:32.171648 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:32.171607    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" path="/var/lib/kubelet/pods/ee52503f-b72e-42fe-a5b2-0a287941daca/volumes"
Nov 02 14:40:32.716583 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:32.716544    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:40:32.716764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:32.716605    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:40:34.756176 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:34.756134    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"]
Nov 02 14:40:34.756716 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:34.756424    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerName="istio-proxy" containerID="cri-o://7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6" gracePeriod=30
Nov 02 14:40:35.084140 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.084106    6284 generic.go:356] "Generic (PLEG): container finished" podID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerID="256e06abd4347c3c130af60ec73300901e8c9c7f5903cdde0dfedfeba3448d02" exitCode=0
Nov 02 14:40:35.084331 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.084184    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" event={"ID":"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183","Type":"ContainerDied","Data":"256e06abd4347c3c130af60ec73300901e8c9c7f5903cdde0dfedfeba3448d02"}
Nov 02 14:40:35.253054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.253026    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:40:35.310821 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.310734    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-kube-api-access-bpmvw\" (UniqueName: \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-kube-api-access-bpmvw\") pod \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") "
Nov 02 14:40:35.311065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.310867    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-envoy\") pod \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") "
Nov 02 14:40:35.311065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.310919    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-podinfo\") pod \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") "
Nov 02 14:40:35.311065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.310992    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-socket\") pod \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") "
Nov 02 14:40:35.311065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.311061    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-token\" (UniqueName: \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-token\") pod \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") "
Nov 02 14:40:35.311316 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.311164    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-certs\") pod \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") "
Nov 02 14:40:35.311316 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.311206    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-data\") pod \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") "
Nov 02 14:40:35.311316 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.311286    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istiod-ca-cert\") pod \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") "
Nov 02 14:40:35.311485 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.311350    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-credential-socket\") pod \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\" (UID: \"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183\") "
Nov 02 14:40:35.311805 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.311778    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-certs" pod "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" (UID: "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:35.312211 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.312192    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:35.312388 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.312361    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-data" pod "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" (UID: "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:35.312524 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.312493    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istiod-ca-cert" pod "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" (UID: "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:40:35.312613 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.312591    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-socket" pod "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" (UID: "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:35.312745 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.312721    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-credential-socket" pod "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" (UID: "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:35.313832 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.313800    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-kube-api-access-bpmvw" pod "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" (UID: "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183"). InnerVolumeSpecName "kube-api-access-bpmvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:35.315206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.315174    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-envoy" pod "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" (UID: "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:35.315318 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.315267    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-token" pod "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" (UID: "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:35.315997 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.315974    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-podinfo" pod "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" (UID: "c76d8ed3-efd1-4e52-b0f8-ba73a77e0183"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:40:35.415396 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.415360    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpmvw\" (UniqueName: \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-kube-api-access-bpmvw\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:35.415396 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.415398    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:35.415810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.415413    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:35.415810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.415421    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:35.415810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.415430    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:35.415810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.415438    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:35.415810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.415445    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:35.415810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:35.415456    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:36.094432 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:36.094398    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"
Nov 02 14:40:36.094432 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:36.094413    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc" event={"ID":"c76d8ed3-efd1-4e52-b0f8-ba73a77e0183","Type":"ContainerDied","Data":"6a0c817361acaef02164c278a741c7febee76972c9440a269698c303137b9ae8"}
Nov 02 14:40:36.095007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:36.094479    6284 scope.go:122] "RemoveContainer" containerID="256e06abd4347c3c130af60ec73300901e8c9c7f5903cdde0dfedfeba3448d02"
Nov 02 14:40:36.159799 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:36.159754    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"]
Nov 02 14:40:36.173142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:36.173108    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-fsslc"]
Nov 02 14:40:38.173024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:38.172983    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" path="/var/lib/kubelet/pods/c76d8ed3-efd1-4e52-b0f8-ba73a77e0183/volumes"
Nov 02 14:40:40.103215 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.103180    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:40:40.130189 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.130147    6284 generic.go:356] "Generic (PLEG): container finished" podID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerID="7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6" exitCode=0
Nov 02 14:40:40.130410 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.130206    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"
Nov 02 14:40:40.130410 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.130229    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" event={"ID":"ca18629f-b465-4a85-87e3-bfc27af3e06a","Type":"ContainerDied","Data":"7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6"}
Nov 02 14:40:40.130410 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.130295    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r" event={"ID":"ca18629f-b465-4a85-87e3-bfc27af3e06a","Type":"ContainerDied","Data":"e9e5636eab6c4794df963fdc8826a346e53fa7d3aee7e405272a477dcd16c468"}
Nov 02 14:40:40.130410 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.130325    6284 scope.go:122] "RemoveContainer" containerID="7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6"
Nov 02 14:40:40.143778 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.143704    6284 scope.go:122] "RemoveContainer" containerID="7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6"
Nov 02 14:40:40.144019 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:40:40.143993    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6\": container with ID starting with 7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6 not found: ID does not exist" containerID="7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6"
Nov 02 14:40:40.144142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.144032    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6"} err="failed to get container status \"7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6\": rpc error: code = NotFound desc = could not find container \"7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6\": container with ID starting with 7e7dcb315883c4a81ee851136cc6d66ac9b4ef7f6803146b5181b2e2263a2da6 not found: ID does not exist"
Nov 02 14:40:40.199910 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.199872    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-credential-socket\") pod \"ca18629f-b465-4a85-87e3-bfc27af3e06a\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") "
Nov 02 14:40:40.200141 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.200110    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-token\" (UniqueName: \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-token\") pod \"ca18629f-b465-4a85-87e3-bfc27af3e06a\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") "
Nov 02 14:40:40.200225 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.200211    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-socket\") pod \"ca18629f-b465-4a85-87e3-bfc27af3e06a\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") "
Nov 02 14:40:40.200286 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.200275    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca18629f-b465-4a85-87e3-bfc27af3e06a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca18629f-b465-4a85-87e3-bfc27af3e06a-istiod-ca-cert\") pod \"ca18629f-b465-4a85-87e3-bfc27af3e06a\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") "
Nov 02 14:40:40.200343 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.200324    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-podinfo\") pod \"ca18629f-b465-4a85-87e3-bfc27af3e06a\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") "
Nov 02 14:40:40.200398 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.200354    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-kube-api-access-nwmr5\" (UniqueName: \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-kube-api-access-nwmr5\") pod \"ca18629f-b465-4a85-87e3-bfc27af3e06a\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") "
Nov 02 14:40:40.200398 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.200388    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-data\") pod \"ca18629f-b465-4a85-87e3-bfc27af3e06a\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") "
Nov 02 14:40:40.200562 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.200546    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-certs\") pod \"ca18629f-b465-4a85-87e3-bfc27af3e06a\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") "
Nov 02 14:40:40.201098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.200818    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-socket" pod "ca18629f-b465-4a85-87e3-bfc27af3e06a" (UID: "ca18629f-b465-4a85-87e3-bfc27af3e06a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:40.201098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.201054    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca18629f-b465-4a85-87e3-bfc27af3e06a-istiod-ca-cert" pod "ca18629f-b465-4a85-87e3-bfc27af3e06a" (UID: "ca18629f-b465-4a85-87e3-bfc27af3e06a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:40:40.201366 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.201324    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-data" pod "ca18629f-b465-4a85-87e3-bfc27af3e06a" (UID: "ca18629f-b465-4a85-87e3-bfc27af3e06a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:40.201500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.201405    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-envoy\") pod \"ca18629f-b465-4a85-87e3-bfc27af3e06a\" (UID: \"ca18629f-b465-4a85-87e3-bfc27af3e06a\") "
Nov 02 14:40:40.202070 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.201769    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-certs" pod "ca18629f-b465-4a85-87e3-bfc27af3e06a" (UID: "ca18629f-b465-4a85-87e3-bfc27af3e06a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:40.203128 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.203105    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca18629f-b465-4a85-87e3-bfc27af3e06a-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:40.203242 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.203134    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:40.203242 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.203147    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:40.203242 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.203162    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:40.204029 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.204004    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-kube-api-access-nwmr5" pod "ca18629f-b465-4a85-87e3-bfc27af3e06a" (UID: "ca18629f-b465-4a85-87e3-bfc27af3e06a"). InnerVolumeSpecName "kube-api-access-nwmr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:40.204576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.204528    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-credential-socket" pod "ca18629f-b465-4a85-87e3-bfc27af3e06a" (UID: "ca18629f-b465-4a85-87e3-bfc27af3e06a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:40.204713 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.204666    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-envoy" pod "ca18629f-b465-4a85-87e3-bfc27af3e06a" (UID: "ca18629f-b465-4a85-87e3-bfc27af3e06a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:40.205112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.205089    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-token" pod "ca18629f-b465-4a85-87e3-bfc27af3e06a" (UID: "ca18629f-b465-4a85-87e3-bfc27af3e06a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:40.207087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.207046    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-podinfo" pod "ca18629f-b465-4a85-87e3-bfc27af3e06a" (UID: "ca18629f-b465-4a85-87e3-bfc27af3e06a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:40:40.304770 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.304730    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:40.304770 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.304765    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca18629f-b465-4a85-87e3-bfc27af3e06a-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:40.304770 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.304774    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:40.304770 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.304784    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca18629f-b465-4a85-87e3-bfc27af3e06a-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:40.305111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.304793    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwmr5\" (UniqueName: \"kubernetes.io/projected/ca18629f-b465-4a85-87e3-bfc27af3e06a-kube-api-access-nwmr5\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:40:40.474518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.474476    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"]
Nov 02 14:40:40.481685 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:40.481645    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-dfj7r"]
Nov 02 14:40:42.173705 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:42.173664    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" path="/var/lib/kubelet/pods/ca18629f-b465-4a85-87e3-bfc27af3e06a/volumes"
Nov 02 14:40:43.897780 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:40:43.897739    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:40:52.475062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.475022    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"]
Nov 02 14:40:52.475922 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.475893    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerName="istio-proxy"
Nov 02 14:40:52.475922 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.475920    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerName="istio-proxy"
Nov 02 14:40:52.476176 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.475977    6284 cpu_manager.go:414] "containerMap: removing container" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerName="istio-proxy"
Nov 02 14:40:52.476176 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.475984    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerName="istio-proxy"
Nov 02 14:40:52.476176 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.476017    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerName="istio-proxy"
Nov 02 14:40:52.476176 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.476022    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerName="istio-proxy"
Nov 02 14:40:52.476176 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.476157    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee52503f-b72e-42fe-a5b2-0a287941daca" containerName="istio-proxy"
Nov 02 14:40:52.476176 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.476178    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76d8ed3-efd1-4e52-b0f8-ba73a77e0183" containerName="istio-proxy"
Nov 02 14:40:52.476384 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.476197    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca18629f-b465-4a85-87e3-bfc27af3e06a" containerName="istio-proxy"
Nov 02 14:40:52.480541 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.480516    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.498331 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.498291    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ea3e8f-serving-cert\""
Nov 02 14:40:52.513795 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.513754    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"]
Nov 02 14:40:52.530000 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.529948    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwqbt\" (UniqueName: \"kubernetes.io/projected/afd5fc01-bcca-445f-8779-561d8aea2504-kube-api-access-nwqbt\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.530213 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.530085    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.530213 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.530153    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd5fc01-bcca-445f-8779-561d8aea2504-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.530213 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.530207    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-metrics-certs\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.631277 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.631236    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqbt\" (UniqueName: \"kubernetes.io/projected/afd5fc01-bcca-445f-8779-561d8aea2504-kube-api-access-nwqbt\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.631469 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.631352    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.631469 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.631413    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd5fc01-bcca-445f-8779-561d8aea2504-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.631469 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.631462    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-metrics-certs\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.631590 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:40:52.631497    6284 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-serving-cert: secret "maas-api-e2e-inf-b-ea3e8f-serving-cert" not found
Nov 02 14:40:52.631590 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:40:52.631581    6284 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-maas-api-tls podName:afd5fc01-bcca-445f-8779-561d8aea2504 nodeName:}" failed. No retries permitted until 2026-11-02 14:40:53.131560572 +0000 UTC m=+3385.090936382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-maas-api-tls") pod "maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82" (UID: "afd5fc01-bcca-445f-8779-561d8aea2504") : secret "maas-api-e2e-inf-b-ea3e8f-serving-cert" not found
Nov 02 14:40:52.632153 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.632132    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd5fc01-bcca-445f-8779-561d8aea2504-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.634113 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.634088    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-metrics-certs\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:52.654292 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:52.654258    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqbt\" (UniqueName: \"kubernetes.io/projected/afd5fc01-bcca-445f-8779-561d8aea2504-kube-api-access-nwqbt\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:53.143203 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:53.143160    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:53.145723 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:53.145696    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:53.397325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:53.397231    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:53.784454 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:53.784414    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"]
Nov 02 14:40:53.808899 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:40:53.808865    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd5fc01_bcca_445f_8779_561d8aea2504.slice/crio-d890162de7b5ebe32cd18a8ea48c090d8c3b1733a636ace709164c49a941a5bb WatchSource:0}: Error finding container d890162de7b5ebe32cd18a8ea48c090d8c3b1733a636ace709164c49a941a5bb: Status 404 returned error can't find the container with id d890162de7b5ebe32cd18a8ea48c090d8c3b1733a636ace709164c49a941a5bb
Nov 02 14:40:53.810970 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:53.810946    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:40:54.257649 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:54.257608    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82" event={"ID":"afd5fc01-bcca-445f-8779-561d8aea2504","Type":"ContainerStarted","Data":"3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f"}
Nov 02 14:40:54.258040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:54.257670    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82" event={"ID":"afd5fc01-bcca-445f-8779-561d8aea2504","Type":"ContainerStarted","Data":"d890162de7b5ebe32cd18a8ea48c090d8c3b1733a636ace709164c49a941a5bb"}
Nov 02 14:40:54.258040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:54.257875    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:40:54.324049 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:40:54.323991    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82" podStartSLOduration=1.9840179070000001 podStartE2EDuration="2.32397444s" totalImagesPullingTime="339.956533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:40:54.322135656 +0000 UTC m=+3386.281511264" watchObservedRunningTime="2026-11-02 14:40:54.32397444 +0000 UTC m=+3386.283350046"
Nov 02 14:41:00.272207 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:00.272177    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:41:09.642435 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.642399    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"]
Nov 02 14:41:09.647137 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.647111    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.670196 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.670140    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cbde78\"/\"kube-root-ca.crt\""
Nov 02 14:41:09.670428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.670355    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cbde78\"/\"openshift-service-ca.crt\""
Nov 02 14:41:09.670687 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.670659    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cbde78\"/\"default-dockercfg-8fqm8\""
Nov 02 14:41:09.675303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.675265    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cbde78\"/\"test-model-cbde78-kserve-self-signed-certs\""
Nov 02 14:41:09.678580 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.678534    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"]
Nov 02 14:41:09.787640 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.787599    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-dshm\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.787859 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.787830    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-model-cache\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.788079 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.788055    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566c48d2-626a-469d-8dd5-25052bf31cf3-tls-certs\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.788135 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.788119    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-tmp-dir\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.788229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.788205    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-home\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.788276 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.788251    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjmbl\" (UniqueName: \"kubernetes.io/projected/566c48d2-626a-469d-8dd5-25052bf31cf3-kube-api-access-pjmbl\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.899024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.896141    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-model-cache\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.899024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.896299    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566c48d2-626a-469d-8dd5-25052bf31cf3-tls-certs\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.899024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.896349    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-tmp-dir\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.899024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.896390    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-home\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.899024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.896420    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjmbl\" (UniqueName: \"kubernetes.io/projected/566c48d2-626a-469d-8dd5-25052bf31cf3-kube-api-access-pjmbl\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.899024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.896591    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-dshm\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.899923 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.899892    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-tmp-dir\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.900611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.900270    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-home\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.900611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.900330    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-model-cache\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.926407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.906995    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-dshm\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.926407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.924287    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566c48d2-626a-469d-8dd5-25052bf31cf3-tls-certs\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.955968 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.948721    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjmbl\" (UniqueName: \"kubernetes.io/projected/566c48d2-626a-469d-8dd5-25052bf31cf3-kube-api-access-pjmbl\") pod \"test-model-cbde78-kserve-865689d97c-57ml8\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") " pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:09.975567 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:09.975254    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:10.328535 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:10.328496    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"]
Nov 02 14:41:10.330036 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:41:10.330006    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod566c48d2_626a_469d_8dd5_25052bf31cf3.slice/crio-ad8eb704588167fe50b0e886825a895011b4e51feda02589a51ea2cb09f00d1b WatchSource:0}: Error finding container ad8eb704588167fe50b0e886825a895011b4e51feda02589a51ea2cb09f00d1b: Status 404 returned error can't find the container with id ad8eb704588167fe50b0e886825a895011b4e51feda02589a51ea2cb09f00d1b
Nov 02 14:41:10.408308 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:10.408270    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8" event={"ID":"566c48d2-626a-469d-8dd5-25052bf31cf3","Type":"ContainerStarted","Data":"ad8eb704588167fe50b0e886825a895011b4e51feda02589a51ea2cb09f00d1b"}
Nov 02 14:41:11.420029 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:11.419986    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8" event={"ID":"566c48d2-626a-469d-8dd5-25052bf31cf3","Type":"ContainerStarted","Data":"6071d1ed846bb2ecd8fc6f15e04e8a39ed07eed7232e65db9fb66eebbcd7a764"}
Nov 02 14:41:11.460422 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:11.460355    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8" podStartSLOduration=2.460336004 podStartE2EDuration="2.460336004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:41:11.456363646 +0000 UTC m=+3403.415739254" watchObservedRunningTime="2026-11-02 14:41:11.460336004 +0000 UTC m=+3403.419711660"
Nov 02 14:41:11.976130 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:11.976091    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:13.896069 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:41:13.896015    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:41:19.976504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:19.976471    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:19.989647 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:19.989622    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:20.515426 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:20.515390    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:41:43.896578 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:41:43.896536    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:41:46.411098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.411058    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"]
Nov 02 14:41:46.415594 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.415572    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.431361 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.431330    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-80133f-serving-cert\""
Nov 02 14:41:46.456593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.456556    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"]
Nov 02 14:41:46.592172 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.592118    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.592373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.592185    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw4pc\" (UniqueName: \"kubernetes.io/projected/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-kube-api-access-jw4pc\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.592373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.592339    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.592477 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.592460    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-metrics-certs\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.708970 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.699212    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.708970 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.699419    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-metrics-certs\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.708970 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.699656    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.708970 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.699705    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw4pc\" (UniqueName: \"kubernetes.io/projected/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-kube-api-access-jw4pc\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.711267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.711230    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.714204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.714173    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.730195 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.730152    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-metrics-certs\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:46.739721 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:46.739686    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw4pc\" (UniqueName: \"kubernetes.io/projected/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-kube-api-access-jw4pc\") pod \"maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:47.031586 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:47.031541    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:47.346433 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:47.346342    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"]
Nov 02 14:41:47.350982 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:41:47.350865    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c1fd3b_8123_4a2a_acc9_c5a45b46d3d9.slice/crio-fd4526f990956920a390ead70d5c083201c751e42ef718abcb9bea1b7231740c WatchSource:0}: Error finding container fd4526f990956920a390ead70d5c083201c751e42ef718abcb9bea1b7231740c: Status 404 returned error can't find the container with id fd4526f990956920a390ead70d5c083201c751e42ef718abcb9bea1b7231740c
Nov 02 14:41:47.804118 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:47.804072    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7" event={"ID":"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9","Type":"ContainerStarted","Data":"fd4526f990956920a390ead70d5c083201c751e42ef718abcb9bea1b7231740c"}
Nov 02 14:41:48.813416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:48.813375    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7" event={"ID":"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9","Type":"ContainerStarted","Data":"d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827"}
Nov 02 14:41:48.813964 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:48.813452    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:41:48.855659 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:48.855596    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7" podStartSLOduration=2.40069563 podStartE2EDuration="2.85557603s" totalImagesPullingTime="454.8804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:41:48.85265061 +0000 UTC m=+3440.812026219" watchObservedRunningTime="2026-11-02 14:41:48.85557603 +0000 UTC m=+3440.814951637"
Nov 02 14:41:54.827024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:41:54.826990    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:42:07.686267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.686220    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"]
Nov 02 14:42:07.692232 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.692201    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.706854 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.706811    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-16c822\"/\"kube-root-ca.crt\""
Nov 02 14:42:07.706854 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.706810    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-16c822\"/\"openshift-service-ca.crt\""
Nov 02 14:42:07.723627 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.723553    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"]
Nov 02 14:42:07.742298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.742263    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-16c822\"/\"test-model-16c822-kserve-self-signed-certs\""
Nov 02 14:42:07.742518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.742396    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-16c822\"/\"default-dockercfg-wmjpl\""
Nov 02 14:42:07.872919 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.872866    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gczwl\" (UniqueName: \"kubernetes.io/projected/e5764355-3c40-4070-a6b1-54498b58ec46-kube-api-access-gczwl\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.873147 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.872924    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-model-cache\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.873147 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.873056    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-home\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.873241 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.873153    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-tmp-dir\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.873356 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.873332    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-dshm\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.873406 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.873396    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5764355-3c40-4070-a6b1-54498b58ec46-tls-certs\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.991223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.990251    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5764355-3c40-4070-a6b1-54498b58ec46-tls-certs\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.991223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.990455    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gczwl\" (UniqueName: \"kubernetes.io/projected/e5764355-3c40-4070-a6b1-54498b58ec46-kube-api-access-gczwl\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.991223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.990513    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-model-cache\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.991223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.990560    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-home\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.991223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.990592    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-tmp-dir\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.991223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.990763    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-dshm\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.991794 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.991282    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-home\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.991794 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.991583    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-model-cache\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.991794 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.991619    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-tmp-dir\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.997213 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.997171    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5764355-3c40-4070-a6b1-54498b58ec46-tls-certs\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:07.999142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:07.998869    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-dshm\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:08.019363 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:08.019315    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gczwl\" (UniqueName: \"kubernetes.io/projected/e5764355-3c40-4070-a6b1-54498b58ec46-kube-api-access-gczwl\") pod \"test-model-16c822-kserve-76fb78b79f-lvffk\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") " pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:08.307437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:08.307334    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:08.635083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:08.635007    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"]
Nov 02 14:42:08.637429 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:42:08.637396    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5764355_3c40_4070_a6b1_54498b58ec46.slice/crio-936e91b2c0a018e72bc024d760080ace8dc1d6ba68ed814b96ddab1ea1e536f5 WatchSource:0}: Error finding container 936e91b2c0a018e72bc024d760080ace8dc1d6ba68ed814b96ddab1ea1e536f5: Status 404 returned error can't find the container with id 936e91b2c0a018e72bc024d760080ace8dc1d6ba68ed814b96ddab1ea1e536f5
Nov 02 14:42:09.010855 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:09.010816    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk" event={"ID":"e5764355-3c40-4070-a6b1-54498b58ec46","Type":"ContainerStarted","Data":"5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d"}
Nov 02 14:42:09.011303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:09.010865    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk" event={"ID":"e5764355-3c40-4070-a6b1-54498b58ec46","Type":"ContainerStarted","Data":"936e91b2c0a018e72bc024d760080ace8dc1d6ba68ed814b96ddab1ea1e536f5"}
Nov 02 14:42:09.055439 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:09.055359    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk" podStartSLOduration=2.05534434 podStartE2EDuration="2.05534434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:42:09.052249301 +0000 UTC m=+3461.011624908" watchObservedRunningTime="2026-11-02 14:42:09.05534434 +0000 UTC m=+3461.014719939"
Nov 02 14:42:09.307930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:09.307814    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:13.897282 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:42:13.897241    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:42:18.307886 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:18.307838    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:18.321464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:18.321432    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:19.111843 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:42:19.111812    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:42:43.896505 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:42:43.896459    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:43:13.896789 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:43:13.896746    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:43:43.897646 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:43:43.897602    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:43:54.857003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:43:54.856956    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"]
Nov 02 14:44:09.663298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:09.663258    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"]
Nov 02 14:44:09.663756 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:09.663487    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy" containerID="cri-o://309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4" gracePeriod=30
Nov 02 14:44:09.672974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:09.672911    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"]
Nov 02 14:44:09.673163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:09.673141    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy" containerID="cri-o://384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d" gracePeriod=30
Nov 02 14:44:09.883255 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:09.883199    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd" podUID="d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" containerName="main" containerID="cri-o://cdc0312627645a11ebfb71c4afbae863f298ac8ce5156acc86ddb716d31b22c0" gracePeriod=45
Nov 02 14:44:10.136873 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.136833    6284 generic.go:356] "Generic (PLEG): container finished" podID="d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" containerID="cdc0312627645a11ebfb71c4afbae863f298ac8ce5156acc86ddb716d31b22c0" exitCode=0
Nov 02 14:44:10.137081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.136881    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd" event={"ID":"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9","Type":"ContainerDied","Data":"cdc0312627645a11ebfb71c4afbae863f298ac8ce5156acc86ddb716d31b22c0"}
Nov 02 14:44:10.166330 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.166301    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:44:10.248759 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.248654    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-kube-api-access-x8t2z\" (UniqueName: \"kubernetes.io/projected/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-kube-api-access-x8t2z\") pod \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") "
Nov 02 14:44:10.248759 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.248717    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tls-certs\") pod \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") "
Nov 02 14:44:10.249290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.248799    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-dshm\") pod \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") "
Nov 02 14:44:10.249290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.248829    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-model-cache\") pod \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") "
Nov 02 14:44:10.249290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.248888    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tmp-dir\") pod \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") "
Nov 02 14:44:10.249290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.249145    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-home\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-home\") pod \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\" (UID: \"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9\") "
Nov 02 14:44:10.250656 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.250616    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-home" pod "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" (UID: "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:10.250656 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.250625    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-model-cache" pod "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" (UID: "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:10.250859 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.250836    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tmp-dir" pod "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" (UID: "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:10.271980 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.253549    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tls-certs" pod "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" (UID: "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:44:10.271980 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.267097    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-dshm" pod "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" (UID: "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:10.273174 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.273140    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-kube-api-access-x8t2z" pod "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" (UID: "d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9"). InnerVolumeSpecName "kube-api-access-x8t2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:10.351808 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.351773    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:10.351808 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.351808    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8t2z\" (UniqueName: \"kubernetes.io/projected/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-kube-api-access-x8t2z\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:10.352049 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.351825    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:10.352049 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.351837    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:10.352049 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.351846    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:10.352049 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.351854    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:10.777052 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.776794    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"]
Nov 02 14:44:10.777580 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:10.777115    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerName="istio-proxy" containerID="cri-o://58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0" gracePeriod=30
Nov 02 14:44:11.146683 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:11.146596    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"
Nov 02 14:44:11.146683 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:11.146612    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd" event={"ID":"d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9","Type":"ContainerDied","Data":"7dad5cffacdc5123f35b73fad0054ebb8a521c61b7aaf782f98b85fbcb27ddcc"}
Nov 02 14:44:11.146683 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:11.146673    6284 scope.go:122] "RemoveContainer" containerID="cdc0312627645a11ebfb71c4afbae863f298ac8ce5156acc86ddb716d31b22c0"
Nov 02 14:44:11.195094 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:11.195049    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"]
Nov 02 14:44:11.199523 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:11.199482    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-09d1a5/test-model-09d1a5-kserve-7cb7bf6b8c-msfkd"]
Nov 02 14:44:12.171334 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:12.171301    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" path="/var/lib/kubelet/pods/d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9/volumes"
Nov 02 14:44:13.187164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:13.183124    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:44:13.187164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:13.183202    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:44:13.897825 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:44:13.897786    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:44:15.010200 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.010171    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:44:15.041491 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.041464    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:44:15.127272 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.127232    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56655b89-7919-4694-8933-fbcc61b81de0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56655b89-7919-4694-8933-fbcc61b81de0-istio-podinfo\") pod \"56655b89-7919-4694-8933-fbcc61b81de0\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") "
Nov 02 14:44:15.127508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.127331    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istiod-ca-cert\") pod \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") "
Nov 02 14:44:15.127508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.127368    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-kube-api-access-78tww\" (UniqueName: \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-kube-api-access-78tww\") pod \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") "
Nov 02 14:44:15.127508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.127417    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-socket\") pod \"56655b89-7919-4694-8933-fbcc61b81de0\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") "
Nov 02 14:44:15.127508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.127460    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-socket\") pod \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") "
Nov 02 14:44:15.127839 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.127781    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-socket" pod "56655b89-7919-4694-8933-fbcc61b81de0" (UID: "56655b89-7919-4694-8933-fbcc61b81de0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.127839 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.127822    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istiod-ca-cert" pod "e68a59a0-b4cf-40e0-9f58-4f0db1436596" (UID: "e68a59a0-b4cf-40e0-9f58-4f0db1436596"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:44:15.127959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.127832    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-socket" pod "e68a59a0-b4cf-40e0-9f58-4f0db1436596" (UID: "e68a59a0-b4cf-40e0-9f58-4f0db1436596"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.128348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.128068    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-certs\") pod \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") "
Nov 02 14:44:15.128348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.128093    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-certs" pod "e68a59a0-b4cf-40e0-9f58-4f0db1436596" (UID: "e68a59a0-b4cf-40e0-9f58-4f0db1436596"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.128348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.128131    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56655b89-7919-4694-8933-fbcc61b81de0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56655b89-7919-4694-8933-fbcc61b81de0-istiod-ca-cert\") pod \"56655b89-7919-4694-8933-fbcc61b81de0\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") "
Nov 02 14:44:15.128348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.128162    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-envoy\") pod \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") "
Nov 02 14:44:15.128348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.128265    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-credential-socket\") pod \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") "
Nov 02 14:44:15.128805 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.128472    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56655b89-7919-4694-8933-fbcc61b81de0-istiod-ca-cert" pod "56655b89-7919-4694-8933-fbcc61b81de0" (UID: "56655b89-7919-4694-8933-fbcc61b81de0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:44:15.128805 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.128726    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-credential-socket" pod "e68a59a0-b4cf-40e0-9f58-4f0db1436596" (UID: "e68a59a0-b4cf-40e0-9f58-4f0db1436596"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.129546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.128318    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-token\" (UniqueName: \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-token\") pod \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") "
Nov 02 14:44:15.129546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.129037    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-envoy\") pod \"56655b89-7919-4694-8933-fbcc61b81de0\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") "
Nov 02 14:44:15.129546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.129115    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-data\") pod \"56655b89-7919-4694-8933-fbcc61b81de0\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") "
Nov 02 14:44:15.129546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.129158    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-podinfo\") pod \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") "
Nov 02 14:44:15.129546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.129188    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-credential-socket\") pod \"56655b89-7919-4694-8933-fbcc61b81de0\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") "
Nov 02 14:44:15.129546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.129226    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-kube-api-access-gx625\" (UniqueName: \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-kube-api-access-gx625\") pod \"56655b89-7919-4694-8933-fbcc61b81de0\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") "
Nov 02 14:44:15.129546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.129325    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-istio-token\" (UniqueName: \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-istio-token\") pod \"56655b89-7919-4694-8933-fbcc61b81de0\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") "
Nov 02 14:44:15.129546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.129377    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-data\") pod \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\" (UID: \"e68a59a0-b4cf-40e0-9f58-4f0db1436596\") "
Nov 02 14:44:15.130014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.129548    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-credential-socket" pod "56655b89-7919-4694-8933-fbcc61b81de0" (UID: "56655b89-7919-4694-8933-fbcc61b81de0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.130014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.129602    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-certs\") pod \"56655b89-7919-4694-8933-fbcc61b81de0\" (UID: \"56655b89-7919-4694-8933-fbcc61b81de0\") "
Nov 02 14:44:15.130014 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.129912    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-data" pod "56655b89-7919-4694-8933-fbcc61b81de0" (UID: "56655b89-7919-4694-8933-fbcc61b81de0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.130244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130189    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-certs" pod "56655b89-7919-4694-8933-fbcc61b81de0" (UID: "56655b89-7919-4694-8933-fbcc61b81de0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.130244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130220    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-data" pod "e68a59a0-b4cf-40e0-9f58-4f0db1436596" (UID: "e68a59a0-b4cf-40e0-9f58-4f0db1436596"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.130610 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130588    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56655b89-7919-4694-8933-fbcc61b81de0-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.130698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130616    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.130698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130633    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.130698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130649    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.130698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130665    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.130698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130678    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.130698 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130691    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.131033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130704    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.131033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130716    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.131033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.130733    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.131491 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.131462    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-kube-api-access-78tww" pod "e68a59a0-b4cf-40e0-9f58-4f0db1436596" (UID: "e68a59a0-b4cf-40e0-9f58-4f0db1436596"). InnerVolumeSpecName "kube-api-access-78tww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:15.132146 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.132120    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56655b89-7919-4694-8933-fbcc61b81de0-istio-podinfo" pod "56655b89-7919-4694-8933-fbcc61b81de0" (UID: "56655b89-7919-4694-8933-fbcc61b81de0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:44:15.133141 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.133103    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-envoy" pod "e68a59a0-b4cf-40e0-9f58-4f0db1436596" (UID: "e68a59a0-b4cf-40e0-9f58-4f0db1436596"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.133318 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.133297    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-token" pod "e68a59a0-b4cf-40e0-9f58-4f0db1436596" (UID: "e68a59a0-b4cf-40e0-9f58-4f0db1436596"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:15.133416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.133333    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-kube-api-access-gx625" pod "56655b89-7919-4694-8933-fbcc61b81de0" (UID: "56655b89-7919-4694-8933-fbcc61b81de0"). InnerVolumeSpecName "kube-api-access-gx625". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:15.134066 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.134041    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-istio-token" pod "56655b89-7919-4694-8933-fbcc61b81de0" (UID: "56655b89-7919-4694-8933-fbcc61b81de0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:15.134303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.134274    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-podinfo" pod "e68a59a0-b4cf-40e0-9f58-4f0db1436596" (UID: "e68a59a0-b4cf-40e0-9f58-4f0db1436596"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:44:15.134784 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.134762    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-envoy" pod "56655b89-7919-4694-8933-fbcc61b81de0" (UID: "56655b89-7919-4694-8933-fbcc61b81de0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.209720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.209622    6284 generic.go:356] "Generic (PLEG): container finished" podID="56655b89-7919-4694-8933-fbcc61b81de0" containerID="309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4" exitCode=0
Nov 02 14:44:15.209720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.209694    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"
Nov 02 14:44:15.209720 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.209705    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" event={"ID":"56655b89-7919-4694-8933-fbcc61b81de0","Type":"ContainerDied","Data":"309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4"}
Nov 02 14:44:15.210042 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.209792    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc" event={"ID":"56655b89-7919-4694-8933-fbcc61b81de0","Type":"ContainerDied","Data":"c49b359bf7eea32eda820ad203d3bd176e90cbebb6e81d4ab57aa1536b72a429"}
Nov 02 14:44:15.210042 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.209813    6284 scope.go:122] "RemoveContainer" containerID="309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4"
Nov 02 14:44:15.211512 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.211487    6284 generic.go:356] "Generic (PLEG): container finished" podID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerID="384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d" exitCode=0
Nov 02 14:44:15.211643 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.211554    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"
Nov 02 14:44:15.211643 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.211618    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" event={"ID":"e68a59a0-b4cf-40e0-9f58-4f0db1436596","Type":"ContainerDied","Data":"384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d"}
Nov 02 14:44:15.211756 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.211686    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8" event={"ID":"e68a59a0-b4cf-40e0-9f58-4f0db1436596","Type":"ContainerDied","Data":"7c52a1a6e697425facfea02596bffec61bb7349e5895318edef5a0ea9d4d384d"}
Nov 02 14:44:15.228905 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.228872    6284 scope.go:122] "RemoveContainer" containerID="309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4"
Nov 02 14:44:15.229314 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:44:15.229291    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4\": container with ID starting with 309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4 not found: ID does not exist" containerID="309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4"
Nov 02 14:44:15.229372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.229324    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4"} err="failed to get container status \"309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4\": rpc error: code = NotFound desc = could not find container \"309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4\": container with ID starting with 309e53c596bc5987234b4e61f6105855f0b4b8107ab95f4ade3b3a7f8ffb73f4 not found: ID does not exist"
Nov 02 14:44:15.229372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.229345    6284 scope.go:122] "RemoveContainer" containerID="384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d"
Nov 02 14:44:15.232858 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.232817    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56655b89-7919-4694-8933-fbcc61b81de0-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.232858 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.232862    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78tww\" (UniqueName: \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-kube-api-access-78tww\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.233117 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.232905    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.233117 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.233006    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.233117 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.233018    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56655b89-7919-4694-8933-fbcc61b81de0-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.233117 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.233039    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e68a59a0-b4cf-40e0-9f58-4f0db1436596-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.233117 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.233055    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx625\" (UniqueName: \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-kube-api-access-gx625\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.233117 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.233071    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56655b89-7919-4694-8933-fbcc61b81de0-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.241058 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.241034    6284 scope.go:122] "RemoveContainer" containerID="384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d"
Nov 02 14:44:15.241439 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:44:15.241417    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d\": container with ID starting with 384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d not found: ID does not exist" containerID="384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d"
Nov 02 14:44:15.241493 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.241451    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d"} err="failed to get container status \"384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d\": rpc error: code = NotFound desc = could not find container \"384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d\": container with ID starting with 384f02d4f19acb02f9b4eaba8bfa03b8178dd818e825fbcef2806122511c000d not found: ID does not exist"
Nov 02 14:44:15.271838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.271804    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"]
Nov 02 14:44:15.275877 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.275839    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-226kc"]
Nov 02 14:44:15.304599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.304553    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"]
Nov 02 14:44:15.308741 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:15.308704    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-c9cq8"]
Nov 02 14:44:16.066632 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.066596    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:44:16.146653 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.146601    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-envoy\") pod \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") "
Nov 02 14:44:16.146653 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.146657    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-podinfo\") pod \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") "
Nov 02 14:44:16.146974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.146701    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-kube-api-access-twpsz\" (UniqueName: \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-kube-api-access-twpsz\") pod \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") "
Nov 02 14:44:16.146974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.146818    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-certs\") pod \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") "
Nov 02 14:44:16.146974 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.146873    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-socket\") pod \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") "
Nov 02 14:44:16.147161 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.147023    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-token\" (UniqueName: \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-token\") pod \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") "
Nov 02 14:44:16.147161 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.147055    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-data\") pod \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") "
Nov 02 14:44:16.147161 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.147121    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-credential-socket\") pod \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") "
Nov 02 14:44:16.147335 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.147199    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istiod-ca-cert\") pod \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\" (UID: \"e0ca6186-3243-459b-ad2a-7d6584ddf0d2\") "
Nov 02 14:44:16.147567 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.147296    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-socket" pod "e0ca6186-3243-459b-ad2a-7d6584ddf0d2" (UID: "e0ca6186-3243-459b-ad2a-7d6584ddf0d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:16.147671 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.147315    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-certs" pod "e0ca6186-3243-459b-ad2a-7d6584ddf0d2" (UID: "e0ca6186-3243-459b-ad2a-7d6584ddf0d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:16.147671 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.147349    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-data" pod "e0ca6186-3243-459b-ad2a-7d6584ddf0d2" (UID: "e0ca6186-3243-459b-ad2a-7d6584ddf0d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:16.147671 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.147504    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-credential-socket" pod "e0ca6186-3243-459b-ad2a-7d6584ddf0d2" (UID: "e0ca6186-3243-459b-ad2a-7d6584ddf0d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:16.147838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.147745    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istiod-ca-cert" pod "e0ca6186-3243-459b-ad2a-7d6584ddf0d2" (UID: "e0ca6186-3243-459b-ad2a-7d6584ddf0d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:44:16.148083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.148055    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.148224 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.148086    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.148224 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.148099    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.148224 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.148114    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.148224 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.148127    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.149276 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.149238    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-envoy" pod "e0ca6186-3243-459b-ad2a-7d6584ddf0d2" (UID: "e0ca6186-3243-459b-ad2a-7d6584ddf0d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:16.149795 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.149762    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-kube-api-access-twpsz" pod "e0ca6186-3243-459b-ad2a-7d6584ddf0d2" (UID: "e0ca6186-3243-459b-ad2a-7d6584ddf0d2"). InnerVolumeSpecName "kube-api-access-twpsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:16.150363 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.150335    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-podinfo" pod "e0ca6186-3243-459b-ad2a-7d6584ddf0d2" (UID: "e0ca6186-3243-459b-ad2a-7d6584ddf0d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:44:16.150604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.150572    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-token" pod "e0ca6186-3243-459b-ad2a-7d6584ddf0d2" (UID: "e0ca6186-3243-459b-ad2a-7d6584ddf0d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:16.177647 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.177600    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56655b89-7919-4694-8933-fbcc61b81de0" path="/var/lib/kubelet/pods/56655b89-7919-4694-8933-fbcc61b81de0/volumes"
Nov 02 14:44:16.178474 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.178449    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" path="/var/lib/kubelet/pods/e68a59a0-b4cf-40e0-9f58-4f0db1436596/volumes"
Nov 02 14:44:16.223588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.223488    6284 generic.go:356] "Generic (PLEG): container finished" podID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerID="58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0" exitCode=0
Nov 02 14:44:16.223588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.223556    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"
Nov 02 14:44:16.223588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.223575    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" event={"ID":"e0ca6186-3243-459b-ad2a-7d6584ddf0d2","Type":"ContainerDied","Data":"58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0"}
Nov 02 14:44:16.223919 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.223626    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9" event={"ID":"e0ca6186-3243-459b-ad2a-7d6584ddf0d2","Type":"ContainerDied","Data":"130d78a063cef022d82d33cba08272d0dd28eeb055c36b6c11ac1bf3b7650788"}
Nov 02 14:44:16.223919 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.223656    6284 scope.go:122] "RemoveContainer" containerID="58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0"
Nov 02 14:44:16.236401 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.236371    6284 scope.go:122] "RemoveContainer" containerID="58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0"
Nov 02 14:44:16.236792 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:44:16.236768    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0\": container with ID starting with 58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0 not found: ID does not exist" containerID="58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0"
Nov 02 14:44:16.236896 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.236807    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0"} err="failed to get container status \"58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0\": rpc error: code = NotFound desc = could not find container \"58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0\": container with ID starting with 58832652fc32d10aa3bc3a300c6b312c3a884cae2190b8a3e5b6d718e40e48d0 not found: ID does not exist"
Nov 02 14:44:16.250019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.249889    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.250019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.249921    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.250019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.249929    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.250019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.249985    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twpsz\" (UniqueName: \"kubernetes.io/projected/e0ca6186-3243-459b-ad2a-7d6584ddf0d2-kube-api-access-twpsz\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.272207 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.272165    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"]
Nov 02 14:44:16.276718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:16.276682    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-65bp9"]
Nov 02 14:44:17.630032 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:17.627477    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"]
Nov 02 14:44:18.172786 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:18.172749    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" path="/var/lib/kubelet/pods/e0ca6186-3243-459b-ad2a-7d6584ddf0d2/volumes"
Nov 02 14:44:19.454018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:19.453976    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"]
Nov 02 14:44:29.148007 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:29.147901    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:44:29.173572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:29.173532    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:44:32.668891 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:32.668841    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9" podUID="ad4dea4c-9cda-4aae-88be-a31096fcb38d" containerName="main" containerID="cri-o://32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e" gracePeriod=45
Nov 02 14:44:32.964751 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:32.964720    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:44:33.130325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.130230    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4dea4c-9cda-4aae-88be-a31096fcb38d-kube-api-access-cfbbl\" (UniqueName: \"kubernetes.io/projected/ad4dea4c-9cda-4aae-88be-a31096fcb38d-kube-api-access-cfbbl\") pod \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") "
Nov 02 14:44:33.130325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.130321    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-home\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-home\") pod \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") "
Nov 02 14:44:33.130593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.130354    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tmp-dir\") pod \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") "
Nov 02 14:44:33.130593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.130402    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-dshm\") pod \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") "
Nov 02 14:44:33.130593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.130451    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-model-cache\") pod \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") "
Nov 02 14:44:33.130593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.130520    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tls-certs\") pod \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\" (UID: \"ad4dea4c-9cda-4aae-88be-a31096fcb38d\") "
Nov 02 14:44:33.130816 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.130606    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-home" pod "ad4dea4c-9cda-4aae-88be-a31096fcb38d" (UID: "ad4dea4c-9cda-4aae-88be-a31096fcb38d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:33.130816 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.130632    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tmp-dir" pod "ad4dea4c-9cda-4aae-88be-a31096fcb38d" (UID: "ad4dea4c-9cda-4aae-88be-a31096fcb38d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:33.130816 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.130789    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-model-cache" pod "ad4dea4c-9cda-4aae-88be-a31096fcb38d" (UID: "ad4dea4c-9cda-4aae-88be-a31096fcb38d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:33.131081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.131059    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:33.131081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.131078    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:33.131269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.131088    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:33.132789 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.132763    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tls-certs" pod "ad4dea4c-9cda-4aae-88be-a31096fcb38d" (UID: "ad4dea4c-9cda-4aae-88be-a31096fcb38d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:44:33.132975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.132787    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4dea4c-9cda-4aae-88be-a31096fcb38d-kube-api-access-cfbbl" pod "ad4dea4c-9cda-4aae-88be-a31096fcb38d" (UID: "ad4dea4c-9cda-4aae-88be-a31096fcb38d"). InnerVolumeSpecName "kube-api-access-cfbbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:33.132975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.132810    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-dshm" pod "ad4dea4c-9cda-4aae-88be-a31096fcb38d" (UID: "ad4dea4c-9cda-4aae-88be-a31096fcb38d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:33.232249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.232200    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4dea4c-9cda-4aae-88be-a31096fcb38d-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:33.232249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.232237    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4dea4c-9cda-4aae-88be-a31096fcb38d-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:33.232249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.232248    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfbbl\" (UniqueName: \"kubernetes.io/projected/ad4dea4c-9cda-4aae-88be-a31096fcb38d-kube-api-access-cfbbl\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:33.373412 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.373378    6284 generic.go:356] "Generic (PLEG): container finished" podID="ad4dea4c-9cda-4aae-88be-a31096fcb38d" containerID="32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e" exitCode=0
Nov 02 14:44:33.373614 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.373430    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"
Nov 02 14:44:33.373614 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.373471    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9" event={"ID":"ad4dea4c-9cda-4aae-88be-a31096fcb38d","Type":"ContainerDied","Data":"32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e"}
Nov 02 14:44:33.373614 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.373517    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9" event={"ID":"ad4dea4c-9cda-4aae-88be-a31096fcb38d","Type":"ContainerDied","Data":"6c09281765413399ce758621da053524ef7a242e535f3c8526a20d0fea705389"}
Nov 02 14:44:33.373614 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.373538    6284 scope.go:122] "RemoveContainer" containerID="32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e"
Nov 02 14:44:33.385768 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.385737    6284 scope.go:122] "RemoveContainer" containerID="32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e"
Nov 02 14:44:33.386124 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:44:33.386101    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e\": container with ID starting with 32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e not found: ID does not exist" containerID="32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e"
Nov 02 14:44:33.386191 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.386134    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e"} err="failed to get container status \"32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e\": rpc error: code = NotFound desc = could not find container \"32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e\": container with ID starting with 32c0e539e292e68550dd39cf1901d7722485eaeb7374e95b9032b9c02506d55e not found: ID does not exist"
Nov 02 14:44:33.425862 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.425824    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"]
Nov 02 14:44:33.432504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:33.432468    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e1fcb4/test-model-e1fcb4-kserve-8594fc84df-x5zd9"]
Nov 02 14:44:34.171163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.171131    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4dea4c-9cda-4aae-88be-a31096fcb38d" path="/var/lib/kubelet/pods/ad4dea4c-9cda-4aae-88be-a31096fcb38d/volumes"
Nov 02 14:44:34.490147 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.490094    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22" podUID="05adda46-8560-440c-a0f6-8ae9252bb27a" containerName="main" containerID="cri-o://1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79" gracePeriod=45
Nov 02 14:44:34.805625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.805595    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:44:34.955848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.955812    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-dshm\") pod \"05adda46-8560-440c-a0f6-8ae9252bb27a\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") "
Nov 02 14:44:34.956059 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.955903    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05adda46-8560-440c-a0f6-8ae9252bb27a-tls-certs\" (UniqueName: \"kubernetes.io/secret/05adda46-8560-440c-a0f6-8ae9252bb27a-tls-certs\") pod \"05adda46-8560-440c-a0f6-8ae9252bb27a\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") "
Nov 02 14:44:34.956059 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.955930    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-tmp-dir\") pod \"05adda46-8560-440c-a0f6-8ae9252bb27a\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") "
Nov 02 14:44:34.956059 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.956030    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05adda46-8560-440c-a0f6-8ae9252bb27a-kube-api-access-xxrg6\" (UniqueName: \"kubernetes.io/projected/05adda46-8560-440c-a0f6-8ae9252bb27a-kube-api-access-xxrg6\") pod \"05adda46-8560-440c-a0f6-8ae9252bb27a\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") "
Nov 02 14:44:34.956179 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.956078    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-home\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-home\") pod \"05adda46-8560-440c-a0f6-8ae9252bb27a\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") "
Nov 02 14:44:34.956179 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.956103    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-model-cache\") pod \"05adda46-8560-440c-a0f6-8ae9252bb27a\" (UID: \"05adda46-8560-440c-a0f6-8ae9252bb27a\") "
Nov 02 14:44:34.956344 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.956312    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-tmp-dir" pod "05adda46-8560-440c-a0f6-8ae9252bb27a" (UID: "05adda46-8560-440c-a0f6-8ae9252bb27a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:34.956432 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.956348    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-home" pod "05adda46-8560-440c-a0f6-8ae9252bb27a" (UID: "05adda46-8560-440c-a0f6-8ae9252bb27a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:34.956495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.956449    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-model-cache" pod "05adda46-8560-440c-a0f6-8ae9252bb27a" (UID: "05adda46-8560-440c-a0f6-8ae9252bb27a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:34.956734 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.956714    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:34.956776 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.956739    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:34.956776 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.956751    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:34.958386 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.958361    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05adda46-8560-440c-a0f6-8ae9252bb27a-kube-api-access-xxrg6" pod "05adda46-8560-440c-a0f6-8ae9252bb27a" (UID: "05adda46-8560-440c-a0f6-8ae9252bb27a"). InnerVolumeSpecName "kube-api-access-xxrg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:34.958658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.958632    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-dshm" pod "05adda46-8560-440c-a0f6-8ae9252bb27a" (UID: "05adda46-8560-440c-a0f6-8ae9252bb27a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:34.958658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:34.958653    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05adda46-8560-440c-a0f6-8ae9252bb27a-tls-certs" pod "05adda46-8560-440c-a0f6-8ae9252bb27a" (UID: "05adda46-8560-440c-a0f6-8ae9252bb27a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:44:35.058477 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.058375    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05adda46-8560-440c-a0f6-8ae9252bb27a-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:35.058477 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.058418    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05adda46-8560-440c-a0f6-8ae9252bb27a-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:35.058477 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.058430    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxrg6\" (UniqueName: \"kubernetes.io/projected/05adda46-8560-440c-a0f6-8ae9252bb27a-kube-api-access-xxrg6\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:44:35.391239 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.391136    6284 generic.go:356] "Generic (PLEG): container finished" podID="05adda46-8560-440c-a0f6-8ae9252bb27a" containerID="1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79" exitCode=0
Nov 02 14:44:35.391239 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.391203    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"
Nov 02 14:44:35.391783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.391225    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22" event={"ID":"05adda46-8560-440c-a0f6-8ae9252bb27a","Type":"ContainerDied","Data":"1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79"}
Nov 02 14:44:35.391783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.391286    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22" event={"ID":"05adda46-8560-440c-a0f6-8ae9252bb27a","Type":"ContainerDied","Data":"3c36a63161ba8746d1a2c2c1d1dce04909e764529d6542775b88aea5125ae11e"}
Nov 02 14:44:35.391783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.391311    6284 scope.go:122] "RemoveContainer" containerID="1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79"
Nov 02 14:44:35.404647 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.404622    6284 scope.go:122] "RemoveContainer" containerID="1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79"
Nov 02 14:44:35.404999 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:44:35.404971    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79\": container with ID starting with 1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79 not found: ID does not exist" containerID="1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79"
Nov 02 14:44:35.405106 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.405008    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79"} err="failed to get container status \"1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79\": rpc error: code = NotFound desc = could not find container \"1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79\": container with ID starting with 1443aa301fff2a4bf5ea2bb724a0c082a7ef3aca170d1c6ab22fae13f0c38a79 not found: ID does not exist"
Nov 02 14:44:35.463491 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.463449    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"]
Nov 02 14:44:35.481033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:35.480998    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d39fa6/test-model-d39fa6-kserve-7b6dc848df-skf22"]
Nov 02 14:44:36.171180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:44:36.171143    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05adda46-8560-440c-a0f6-8ae9252bb27a" path="/var/lib/kubelet/pods/05adda46-8560-440c-a0f6-8ae9252bb27a/volumes"
Nov 02 14:44:43.896230 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:44:43.896179    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:45:07.009177 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:07.009130    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"]
Nov 02 14:45:07.009822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:07.009418    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy" containerID="cri-o://bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a" gracePeriod=30
Nov 02 14:45:07.470407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:07.470323    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:45:07.470407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:07.470382    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:45:12.304009 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.303976    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 14:45:12.387115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387065    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-data\") pod \"94cc467e-61a3-4f4b-b317-60fe7deffb63\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") "
Nov 02 14:45:12.387321 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387195    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-credential-socket\") pod \"94cc467e-61a3-4f4b-b317-60fe7deffb63\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") "
Nov 02 14:45:12.387321 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387234    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-envoy\") pod \"94cc467e-61a3-4f4b-b317-60fe7deffb63\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") "
Nov 02 14:45:12.387420 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387378    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-podinfo\") pod \"94cc467e-61a3-4f4b-b317-60fe7deffb63\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") "
Nov 02 14:45:12.387472 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387408    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-data" pod "94cc467e-61a3-4f4b-b317-60fe7deffb63" (UID: "94cc467e-61a3-4f4b-b317-60fe7deffb63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:12.387472 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387419    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-certs\") pod \"94cc467e-61a3-4f4b-b317-60fe7deffb63\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") "
Nov 02 14:45:12.387588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387540    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94cc467e-61a3-4f4b-b317-60fe7deffb63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cc467e-61a3-4f4b-b317-60fe7deffb63-istiod-ca-cert\") pod \"94cc467e-61a3-4f4b-b317-60fe7deffb63\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") "
Nov 02 14:45:12.387692 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387671    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-socket\") pod \"94cc467e-61a3-4f4b-b317-60fe7deffb63\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") "
Nov 02 14:45:12.387759 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387693    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-certs" pod "94cc467e-61a3-4f4b-b317-60fe7deffb63" (UID: "94cc467e-61a3-4f4b-b317-60fe7deffb63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:12.387759 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387711    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-kube-api-access-hqllb\" (UniqueName: \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-kube-api-access-hqllb\") pod \"94cc467e-61a3-4f4b-b317-60fe7deffb63\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") "
Nov 02 14:45:12.387877 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387808    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-token\" (UniqueName: \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-token\") pod \"94cc467e-61a3-4f4b-b317-60fe7deffb63\" (UID: \"94cc467e-61a3-4f4b-b317-60fe7deffb63\") "
Nov 02 14:45:12.387992 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.387922    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-credential-socket" pod "94cc467e-61a3-4f4b-b317-60fe7deffb63" (UID: "94cc467e-61a3-4f4b-b317-60fe7deffb63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:12.388077 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.388054    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94cc467e-61a3-4f4b-b317-60fe7deffb63-istiod-ca-cert" pod "94cc467e-61a3-4f4b-b317-60fe7deffb63" (UID: "94cc467e-61a3-4f4b-b317-60fe7deffb63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:12.388204 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.388178    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-socket" pod "94cc467e-61a3-4f4b-b317-60fe7deffb63" (UID: "94cc467e-61a3-4f4b-b317-60fe7deffb63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:12.388565 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.388541    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:12.388709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.388685    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94cc467e-61a3-4f4b-b317-60fe7deffb63-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:12.388709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.388714    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:12.388888 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.388728    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:12.388888 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.388743    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:12.390353 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.390327    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-envoy" pod "94cc467e-61a3-4f4b-b317-60fe7deffb63" (UID: "94cc467e-61a3-4f4b-b317-60fe7deffb63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:12.390453 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.390336    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-podinfo" pod "94cc467e-61a3-4f4b-b317-60fe7deffb63" (UID: "94cc467e-61a3-4f4b-b317-60fe7deffb63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:45:12.390650 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.390630    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-kube-api-access-hqllb" pod "94cc467e-61a3-4f4b-b317-60fe7deffb63" (UID: "94cc467e-61a3-4f4b-b317-60fe7deffb63"). InnerVolumeSpecName "kube-api-access-hqllb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:12.390993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.390975    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-token" pod "94cc467e-61a3-4f4b-b317-60fe7deffb63" (UID: "94cc467e-61a3-4f4b-b317-60fe7deffb63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:12.489696 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.489655    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:12.489696 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.489693    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqllb\" (UniqueName: \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-kube-api-access-hqllb\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:12.489696 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.489703    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:12.489997 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.489712    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94cc467e-61a3-4f4b-b317-60fe7deffb63-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:12.741471 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.741433    6284 generic.go:356] "Generic (PLEG): container finished" podID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerID="bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a" exitCode=0
Nov 02 14:45:12.741677 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.741486    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"
Nov 02 14:45:12.741677 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.741523    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" event={"ID":"94cc467e-61a3-4f4b-b317-60fe7deffb63","Type":"ContainerDied","Data":"bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a"}
Nov 02 14:45:12.741677 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.741587    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k" event={"ID":"94cc467e-61a3-4f4b-b317-60fe7deffb63","Type":"ContainerDied","Data":"46137e497ae7c317fff790f96ea93578ded0d7bb69098035f50d38b85f57ef7f"}
Nov 02 14:45:12.741677 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.741617    6284 scope.go:122] "RemoveContainer" containerID="bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a"
Nov 02 14:45:12.758600 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.758576    6284 scope.go:122] "RemoveContainer" containerID="bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a"
Nov 02 14:45:12.759052 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:45:12.759022    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a\": container with ID starting with bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a not found: ID does not exist" containerID="bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a"
Nov 02 14:45:12.759170 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.759059    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a"} err="failed to get container status \"bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a\": rpc error: code = NotFound desc = could not find container \"bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a\": container with ID starting with bb6f0bfc2e399a7a2e263e7d54f71970e40dc9f5971b0d05785946365275c75a not found: ID does not exist"
Nov 02 14:45:12.789744 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.789705    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"]
Nov 02 14:45:12.793970 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:12.793918    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-6n72k"]
Nov 02 14:45:13.897228 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:45:13.897178    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:45:14.172465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.172381    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" path="/var/lib/kubelet/pods/94cc467e-61a3-4f4b-b317-60fe7deffb63/volumes"
Nov 02 14:45:14.282101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.282069    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"]
Nov 02 14:45:14.282358 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.282329    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9" podUID="4c1a8118-ecdb-48a7-af76-2c1fe30f71ec" containerName="maas-api" containerID="cri-o://cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed" gracePeriod=30
Nov 02 14:45:14.731325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.731285    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"]
Nov 02 14:45:14.731801 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.731768    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx" podUID="106517f1-b322-462b-80ba-ae2c8316bc62" containerName="maas-api" containerID="cri-o://e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d" gracePeriod=30
Nov 02 14:45:14.757095 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.756283    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:45:14.764442 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.764406    6284 generic.go:356] "Generic (PLEG): container finished" podID="4c1a8118-ecdb-48a7-af76-2c1fe30f71ec" containerID="cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed" exitCode=0
Nov 02 14:45:14.764711 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.764462    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"
Nov 02 14:45:14.764880 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.764478    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9" event={"ID":"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec","Type":"ContainerDied","Data":"cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed"}
Nov 02 14:45:14.765142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.764923    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9" event={"ID":"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec","Type":"ContainerDied","Data":"b6bfe891d8536387ed2cb7cc3f5e47d9f5c5d4fd9fcef5b9e44e1adacf32eff0"}
Nov 02 14:45:14.765236 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.765192    6284 scope.go:122] "RemoveContainer" containerID="cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed"
Nov 02 14:45:14.886789 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.886409    6284 scope.go:122] "RemoveContainer" containerID="cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed"
Nov 02 14:45:14.887038 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:45:14.886916    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed\": container with ID starting with cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed not found: ID does not exist" containerID="cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed"
Nov 02 14:45:14.887038 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.886973    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed"} err="failed to get container status \"cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed\": rpc error: code = NotFound desc = could not find container \"cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed\": container with ID starting with cfd5170dd36f5ccd5d2b994a6e9db23b96241d5b56b075358b048eb2b56c96ed not found: ID does not exist"
Nov 02 14:45:14.952525 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.952485    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-metrics-certs\") pod \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") "
Nov 02 14:45:14.953064 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.952605    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-kube-api-access-7lb2j\" (UniqueName: \"kubernetes.io/projected/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-kube-api-access-7lb2j\") pod \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") "
Nov 02 14:45:14.953064 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.952782    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-maas-api-tls\") pod \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") "
Nov 02 14:45:14.953064 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.952856    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-openshift-service-ca\") pod \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\" (UID: \"4c1a8118-ecdb-48a7-af76-2c1fe30f71ec\") "
Nov 02 14:45:14.954235 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.954184    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-openshift-service-ca" pod "4c1a8118-ecdb-48a7-af76-2c1fe30f71ec" (UID: "4c1a8118-ecdb-48a7-af76-2c1fe30f71ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:14.956871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.956818    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-metrics-certs" pod "4c1a8118-ecdb-48a7-af76-2c1fe30f71ec" (UID: "4c1a8118-ecdb-48a7-af76-2c1fe30f71ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:45:14.957148 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.957122    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-maas-api-tls" pod "4c1a8118-ecdb-48a7-af76-2c1fe30f71ec" (UID: "4c1a8118-ecdb-48a7-af76-2c1fe30f71ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:45:14.958315 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.958274    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-kube-api-access-7lb2j" pod "4c1a8118-ecdb-48a7-af76-2c1fe30f71ec" (UID: "4c1a8118-ecdb-48a7-af76-2c1fe30f71ec"). InnerVolumeSpecName "kube-api-access-7lb2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:14.962978 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:14.962665    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx" podUID="106517f1-b322-462b-80ba-ae2c8316bc62" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.147:8443/health\": dial tcp 10.134.0.147:8443: connect: connection refused"
Nov 02 14:45:15.070098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.068589    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.070098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.068711    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lb2j\" (UniqueName: \"kubernetes.io/projected/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-kube-api-access-7lb2j\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.070098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.068727    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.070098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.068743    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.108040 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.107629    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:45:15.131694 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.131302    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"]
Nov 02 14:45:15.137262 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.137225    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e1fcb4-7968786d7-blfd9"]
Nov 02 14:45:15.179958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.174169    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106517f1-b322-462b-80ba-ae2c8316bc62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/106517f1-b322-462b-80ba-ae2c8316bc62-openshift-service-ca\") pod \"106517f1-b322-462b-80ba-ae2c8316bc62\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") "
Nov 02 14:45:15.179958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.174366    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-maas-api-tls\") pod \"106517f1-b322-462b-80ba-ae2c8316bc62\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") "
Nov 02 14:45:15.179958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.174437    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-metrics-certs\") pod \"106517f1-b322-462b-80ba-ae2c8316bc62\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") "
Nov 02 14:45:15.179958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.174479    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106517f1-b322-462b-80ba-ae2c8316bc62-kube-api-access-c5vcw\" (UniqueName: \"kubernetes.io/projected/106517f1-b322-462b-80ba-ae2c8316bc62-kube-api-access-c5vcw\") pod \"106517f1-b322-462b-80ba-ae2c8316bc62\" (UID: \"106517f1-b322-462b-80ba-ae2c8316bc62\") "
Nov 02 14:45:15.179958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.178328    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106517f1-b322-462b-80ba-ae2c8316bc62-openshift-service-ca" pod "106517f1-b322-462b-80ba-ae2c8316bc62" (UID: "106517f1-b322-462b-80ba-ae2c8316bc62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:15.206001 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.199132    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106517f1-b322-462b-80ba-ae2c8316bc62-kube-api-access-c5vcw" pod "106517f1-b322-462b-80ba-ae2c8316bc62" (UID: "106517f1-b322-462b-80ba-ae2c8316bc62"). InnerVolumeSpecName "kube-api-access-c5vcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:15.206001 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.199266    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-metrics-certs" pod "106517f1-b322-462b-80ba-ae2c8316bc62" (UID: "106517f1-b322-462b-80ba-ae2c8316bc62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:45:15.206001 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.199352    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-maas-api-tls" pod "106517f1-b322-462b-80ba-ae2c8316bc62" (UID: "106517f1-b322-462b-80ba-ae2c8316bc62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:45:15.278593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.276891    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/106517f1-b322-462b-80ba-ae2c8316bc62-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.278593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.276928    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.278593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.276951    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/106517f1-b322-462b-80ba-ae2c8316bc62-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.278593 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.276960    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5vcw\" (UniqueName: \"kubernetes.io/projected/106517f1-b322-462b-80ba-ae2c8316bc62-kube-api-access-c5vcw\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.775882 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.775845    6284 generic.go:356] "Generic (PLEG): container finished" podID="106517f1-b322-462b-80ba-ae2c8316bc62" containerID="e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d" exitCode=0
Nov 02 14:45:15.776112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.775894    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"
Nov 02 14:45:15.776112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.775920    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx" event={"ID":"106517f1-b322-462b-80ba-ae2c8316bc62","Type":"ContainerDied","Data":"e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d"}
Nov 02 14:45:15.776112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.776001    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx" event={"ID":"106517f1-b322-462b-80ba-ae2c8316bc62","Type":"ContainerDied","Data":"a8c73607b239dadb4898026b4e7c9d169bcbc62e926f0552303c46a0adb5bd3f"}
Nov 02 14:45:15.776112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.776034    6284 scope.go:122] "RemoveContainer" containerID="e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d"
Nov 02 14:45:15.787056 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.787033    6284 scope.go:122] "RemoveContainer" containerID="e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d"
Nov 02 14:45:15.787478 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:45:15.787455    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d\": container with ID starting with e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d not found: ID does not exist" containerID="e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d"
Nov 02 14:45:15.787607 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.787487    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d"} err="failed to get container status \"e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d\": rpc error: code = NotFound desc = could not find container \"e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d\": container with ID starting with e5c94f7389c3af1e1d9fb278876c316bf7fa6fe1447a9db64dca2719b6d3913d not found: ID does not exist"
Nov 02 14:45:15.822795 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.822753    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"]
Nov 02 14:45:15.832388 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:15.832354    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-09d1a5-665f5c77b6-4kvsx"]
Nov 02 14:45:16.171913 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:16.171822    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106517f1-b322-462b-80ba-ae2c8316bc62" path="/var/lib/kubelet/pods/106517f1-b322-462b-80ba-ae2c8316bc62/volumes"
Nov 02 14:45:16.172364 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:45:16.172349    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1a8118-ecdb-48a7-af76-2c1fe30f71ec" path="/var/lib/kubelet/pods/4c1a8118-ecdb-48a7-af76-2c1fe30f71ec/volumes"
Nov 02 14:45:43.895615 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:45:43.895570    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:46:07.186533 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:07.186484    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"]
Nov 02 14:46:13.896581 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:46:13.896541    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:46:16.833905 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:16.833859    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"]
Nov 02 14:46:16.834529 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:16.834118    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv" podUID="0059df5c-839a-49af-8f54-2316292dbc49" containerName="maas-api" containerID="cri-o://8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a" gracePeriod=30
Nov 02 14:46:17.192062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.192027    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:46:17.345210 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.344205    6284 generic.go:356] "Generic (PLEG): container finished" podID="0059df5c-839a-49af-8f54-2316292dbc49" containerID="8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a" exitCode=0
Nov 02 14:46:17.345210 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.344415    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv" event={"ID":"0059df5c-839a-49af-8f54-2316292dbc49","Type":"ContainerDied","Data":"8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a"}
Nov 02 14:46:17.345210 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.344492    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv" event={"ID":"0059df5c-839a-49af-8f54-2316292dbc49","Type":"ContainerDied","Data":"ebbb1174e86470a1f60e5765f072d4079a12551991fe07475c249d9d4aafdf2d"}
Nov 02 14:46:17.345210 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.344552    6284 scope.go:122] "RemoveContainer" containerID="8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a"
Nov 02 14:46:17.345210 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.344733    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"
Nov 02 14:46:17.379174 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.374220    6284 scope.go:122] "RemoveContainer" containerID="8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a"
Nov 02 14:46:17.379174 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:46:17.374688    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a\": container with ID starting with 8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a not found: ID does not exist" containerID="8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a"
Nov 02 14:46:17.379174 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.374765    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a"} err="failed to get container status \"8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a\": rpc error: code = NotFound desc = could not find container \"8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a\": container with ID starting with 8009a8bdd9f19892a686f269bc596f7162e4988b0c2e809a4640f9692e738e3a not found: ID does not exist"
Nov 02 14:46:17.448106 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.448000    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-maas-api-tls\") pod \"0059df5c-839a-49af-8f54-2316292dbc49\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") "
Nov 02 14:46:17.448106 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.448085    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0059df5c-839a-49af-8f54-2316292dbc49-kube-api-access-nnq4c\" (UniqueName: \"kubernetes.io/projected/0059df5c-839a-49af-8f54-2316292dbc49-kube-api-access-nnq4c\") pod \"0059df5c-839a-49af-8f54-2316292dbc49\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") "
Nov 02 14:46:17.448350 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.448170    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-metrics-certs\") pod \"0059df5c-839a-49af-8f54-2316292dbc49\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") "
Nov 02 14:46:17.448350 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.448315    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0059df5c-839a-49af-8f54-2316292dbc49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0059df5c-839a-49af-8f54-2316292dbc49-openshift-service-ca\") pod \"0059df5c-839a-49af-8f54-2316292dbc49\" (UID: \"0059df5c-839a-49af-8f54-2316292dbc49\") "
Nov 02 14:46:17.449079 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.449042    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0059df5c-839a-49af-8f54-2316292dbc49-openshift-service-ca" pod "0059df5c-839a-49af-8f54-2316292dbc49" (UID: "0059df5c-839a-49af-8f54-2316292dbc49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:46:17.450895 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.450862    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-maas-api-tls" pod "0059df5c-839a-49af-8f54-2316292dbc49" (UID: "0059df5c-839a-49af-8f54-2316292dbc49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:46:17.451069 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.450893    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0059df5c-839a-49af-8f54-2316292dbc49-kube-api-access-nnq4c" pod "0059df5c-839a-49af-8f54-2316292dbc49" (UID: "0059df5c-839a-49af-8f54-2316292dbc49"). InnerVolumeSpecName "kube-api-access-nnq4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:17.451315 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.451296    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-metrics-certs" pod "0059df5c-839a-49af-8f54-2316292dbc49" (UID: "0059df5c-839a-49af-8f54-2316292dbc49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:46:17.550219 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.550184    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:17.550219 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.550220    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnq4c\" (UniqueName: \"kubernetes.io/projected/0059df5c-839a-49af-8f54-2316292dbc49-kube-api-access-nnq4c\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:17.550464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.550236    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0059df5c-839a-49af-8f54-2316292dbc49-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:17.550464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.550245    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0059df5c-839a-49af-8f54-2316292dbc49-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:17.695510 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.695465    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"]
Nov 02 14:46:17.699817 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:17.699725    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e1136e-754fb478cb-xntcv"]
Nov 02 14:46:18.171446 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:18.171405    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0059df5c-839a-49af-8f54-2316292dbc49" path="/var/lib/kubelet/pods/0059df5c-839a-49af-8f54-2316292dbc49/volumes"
Nov 02 14:46:20.576562 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:20.576529    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"]
Nov 02 14:46:20.577150 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:20.576750    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6" podUID="cf597423-a322-4d66-a75f-6fdda939de05" containerName="maas-api" containerID="cri-o://691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225" gracePeriod=30
Nov 02 14:46:20.950473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:20.950438    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:46:21.107325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.107270    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-metrics-certs\") pod \"cf597423-a322-4d66-a75f-6fdda939de05\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") "
Nov 02 14:46:21.107574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.107384    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf597423-a322-4d66-a75f-6fdda939de05-kube-api-access-cm5lk\" (UniqueName: \"kubernetes.io/projected/cf597423-a322-4d66-a75f-6fdda939de05-kube-api-access-cm5lk\") pod \"cf597423-a322-4d66-a75f-6fdda939de05\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") "
Nov 02 14:46:21.107653 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.107584    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf597423-a322-4d66-a75f-6fdda939de05-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf597423-a322-4d66-a75f-6fdda939de05-openshift-service-ca\") pod \"cf597423-a322-4d66-a75f-6fdda939de05\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") "
Nov 02 14:46:21.108017 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.107987    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf597423-a322-4d66-a75f-6fdda939de05-openshift-service-ca" pod "cf597423-a322-4d66-a75f-6fdda939de05" (UID: "cf597423-a322-4d66-a75f-6fdda939de05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:46:21.108172 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.108058    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-maas-api-tls\") pod \"cf597423-a322-4d66-a75f-6fdda939de05\" (UID: \"cf597423-a322-4d66-a75f-6fdda939de05\") "
Nov 02 14:46:21.108709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.108689    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf597423-a322-4d66-a75f-6fdda939de05-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:21.114295 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.110337    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-metrics-certs" pod "cf597423-a322-4d66-a75f-6fdda939de05" (UID: "cf597423-a322-4d66-a75f-6fdda939de05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:46:21.114295 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.111632    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf597423-a322-4d66-a75f-6fdda939de05-kube-api-access-cm5lk" pod "cf597423-a322-4d66-a75f-6fdda939de05" (UID: "cf597423-a322-4d66-a75f-6fdda939de05"). InnerVolumeSpecName "kube-api-access-cm5lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:21.114295 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.113639    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-maas-api-tls" pod "cf597423-a322-4d66-a75f-6fdda939de05" (UID: "cf597423-a322-4d66-a75f-6fdda939de05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:46:21.210872 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.210778    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:21.210872 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.210815    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm5lk\" (UniqueName: \"kubernetes.io/projected/cf597423-a322-4d66-a75f-6fdda939de05-kube-api-access-cm5lk\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:21.210872 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.210824    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf597423-a322-4d66-a75f-6fdda939de05-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:21.378741 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.378702    6284 generic.go:356] "Generic (PLEG): container finished" podID="cf597423-a322-4d66-a75f-6fdda939de05" containerID="691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225" exitCode=0
Nov 02 14:46:21.378955 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.378751    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"
Nov 02 14:46:21.378955 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.378785    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6" event={"ID":"cf597423-a322-4d66-a75f-6fdda939de05","Type":"ContainerDied","Data":"691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225"}
Nov 02 14:46:21.378955 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.378847    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6" event={"ID":"cf597423-a322-4d66-a75f-6fdda939de05","Type":"ContainerDied","Data":"d27f557d428971a3cc784da9783a96f66cbbc482f9dc2270fa660517a8f05ceb"}
Nov 02 14:46:21.378955 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.378875    6284 scope.go:122] "RemoveContainer" containerID="691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225"
Nov 02 14:46:21.391989 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.391911    6284 scope.go:122] "RemoveContainer" containerID="691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225"
Nov 02 14:46:21.392274 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:46:21.392256    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225\": container with ID starting with 691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225 not found: ID does not exist" containerID="691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225"
Nov 02 14:46:21.392317 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.392284    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225"} err="failed to get container status \"691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225\": rpc error: code = NotFound desc = could not find container \"691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225\": container with ID starting with 691d967ce7758c2522721fc94b13def45e5d29d4535d82c65473b705f3bd7225 not found: ID does not exist"
Nov 02 14:46:21.468881 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.468783    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"]
Nov 02 14:46:21.473905 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:21.473867    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d39fa6-5db7b5f594-5bwh6"]
Nov 02 14:46:22.174702 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.174668    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf597423-a322-4d66-a75f-6fdda939de05" path="/var/lib/kubelet/pods/cf597423-a322-4d66-a75f-6fdda939de05/volumes"
Nov 02 14:46:22.211099 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.211055    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8" podUID="566c48d2-626a-469d-8dd5-25052bf31cf3" containerName="main" containerID="cri-o://6071d1ed846bb2ecd8fc6f15e04e8a39ed07eed7232e65db9fb66eebbcd7a764" gracePeriod=45
Nov 02 14:46:22.395740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.395490    6284 generic.go:356] "Generic (PLEG): container finished" podID="566c48d2-626a-469d-8dd5-25052bf31cf3" containerID="6071d1ed846bb2ecd8fc6f15e04e8a39ed07eed7232e65db9fb66eebbcd7a764" exitCode=0
Nov 02 14:46:22.395740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.395677    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8" event={"ID":"566c48d2-626a-469d-8dd5-25052bf31cf3","Type":"ContainerDied","Data":"6071d1ed846bb2ecd8fc6f15e04e8a39ed07eed7232e65db9fb66eebbcd7a764"}
Nov 02 14:46:22.518202 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.518174    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:46:22.527164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.527127    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/566c48d2-626a-469d-8dd5-25052bf31cf3-kube-api-access-pjmbl\" (UniqueName: \"kubernetes.io/projected/566c48d2-626a-469d-8dd5-25052bf31cf3-kube-api-access-pjmbl\") pod \"566c48d2-626a-469d-8dd5-25052bf31cf3\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") "
Nov 02 14:46:22.527382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.527206    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-home\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-home\") pod \"566c48d2-626a-469d-8dd5-25052bf31cf3\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") "
Nov 02 14:46:22.527382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.527256    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-dshm\") pod \"566c48d2-626a-469d-8dd5-25052bf31cf3\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") "
Nov 02 14:46:22.527382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.527291    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/566c48d2-626a-469d-8dd5-25052bf31cf3-tls-certs\" (UniqueName: \"kubernetes.io/secret/566c48d2-626a-469d-8dd5-25052bf31cf3-tls-certs\") pod \"566c48d2-626a-469d-8dd5-25052bf31cf3\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") "
Nov 02 14:46:22.527382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.527333    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-model-cache\") pod \"566c48d2-626a-469d-8dd5-25052bf31cf3\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") "
Nov 02 14:46:22.527382 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.527376    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-tmp-dir\") pod \"566c48d2-626a-469d-8dd5-25052bf31cf3\" (UID: \"566c48d2-626a-469d-8dd5-25052bf31cf3\") "
Nov 02 14:46:22.527664 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.527507    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-home" pod "566c48d2-626a-469d-8dd5-25052bf31cf3" (UID: "566c48d2-626a-469d-8dd5-25052bf31cf3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:22.527755 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.527729    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-model-cache" pod "566c48d2-626a-469d-8dd5-25052bf31cf3" (UID: "566c48d2-626a-469d-8dd5-25052bf31cf3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:22.527807 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.527738    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-tmp-dir" pod "566c48d2-626a-469d-8dd5-25052bf31cf3" (UID: "566c48d2-626a-469d-8dd5-25052bf31cf3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:22.528337 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.528287    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:22.528337 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.528308    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:22.528337 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.528316    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:22.529992 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.529962    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-dshm" pod "566c48d2-626a-469d-8dd5-25052bf31cf3" (UID: "566c48d2-626a-469d-8dd5-25052bf31cf3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:22.530181 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.530159    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566c48d2-626a-469d-8dd5-25052bf31cf3-kube-api-access-pjmbl" pod "566c48d2-626a-469d-8dd5-25052bf31cf3" (UID: "566c48d2-626a-469d-8dd5-25052bf31cf3"). InnerVolumeSpecName "kube-api-access-pjmbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:22.530240 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.530163    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/566c48d2-626a-469d-8dd5-25052bf31cf3-tls-certs" pod "566c48d2-626a-469d-8dd5-25052bf31cf3" (UID: "566c48d2-626a-469d-8dd5-25052bf31cf3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:46:22.630444 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.630342    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjmbl\" (UniqueName: \"kubernetes.io/projected/566c48d2-626a-469d-8dd5-25052bf31cf3-kube-api-access-pjmbl\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:22.630444 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.630385    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/566c48d2-626a-469d-8dd5-25052bf31cf3-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:22.630444 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:22.630404    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/566c48d2-626a-469d-8dd5-25052bf31cf3-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:23.407003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:23.406974    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"
Nov 02 14:46:23.407515 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:23.406974    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8" event={"ID":"566c48d2-626a-469d-8dd5-25052bf31cf3","Type":"ContainerDied","Data":"ad8eb704588167fe50b0e886825a895011b4e51feda02589a51ea2cb09f00d1b"}
Nov 02 14:46:23.407515 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:23.407114    6284 scope.go:122] "RemoveContainer" containerID="6071d1ed846bb2ecd8fc6f15e04e8a39ed07eed7232e65db9fb66eebbcd7a764"
Nov 02 14:46:23.460327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:23.460280    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"]
Nov 02 14:46:23.470399 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:23.470359    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8"]
Nov 02 14:46:23.503881 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:23.503828    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-cbde78/test-model-cbde78-kserve-865689d97c-57ml8" podUID="566c48d2-626a-469d-8dd5-25052bf31cf3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:46:24.171856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:24.171822    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="566c48d2-626a-469d-8dd5-25052bf31cf3" path="/var/lib/kubelet/pods/566c48d2-626a-469d-8dd5-25052bf31cf3/volumes"
Nov 02 14:46:40.370982 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.370920    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"]
Nov 02 14:46:40.371464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.371227    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v" podUID="440dcea9-b109-41e6-aad5-bf7dcdaf4c7a" containerName="maas-api" containerID="cri-o://e6c6f9cb63bb85fbac76f307c177abf993ce122e43f15e78fd3db36190bb508c" gracePeriod=30
Nov 02 14:46:40.548795 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.548757    6284 generic.go:356] "Generic (PLEG): container finished" podID="440dcea9-b109-41e6-aad5-bf7dcdaf4c7a" containerID="e6c6f9cb63bb85fbac76f307c177abf993ce122e43f15e78fd3db36190bb508c" exitCode=0
Nov 02 14:46:40.549088 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.548828    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v" event={"ID":"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a","Type":"ContainerDied","Data":"e6c6f9cb63bb85fbac76f307c177abf993ce122e43f15e78fd3db36190bb508c"}
Nov 02 14:46:40.715871 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.715841    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:46:40.883147 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.883107    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-metrics-certs\") pod \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") "
Nov 02 14:46:40.883357 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.883186    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-openshift-service-ca\") pod \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") "
Nov 02 14:46:40.883405 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.883364    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-maas-api-tls\") pod \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") "
Nov 02 14:46:40.883464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.883450    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-kube-api-access-cq86l\" (UniqueName: \"kubernetes.io/projected/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-kube-api-access-cq86l\") pod \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\" (UID: \"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a\") "
Nov 02 14:46:40.883558 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.883534    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-openshift-service-ca" pod "440dcea9-b109-41e6-aad5-bf7dcdaf4c7a" (UID: "440dcea9-b109-41e6-aad5-bf7dcdaf4c7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:46:40.883854 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.883834    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:40.885526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.885496    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-maas-api-tls" pod "440dcea9-b109-41e6-aad5-bf7dcdaf4c7a" (UID: "440dcea9-b109-41e6-aad5-bf7dcdaf4c7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:46:40.885724 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.885698    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-metrics-certs" pod "440dcea9-b109-41e6-aad5-bf7dcdaf4c7a" (UID: "440dcea9-b109-41e6-aad5-bf7dcdaf4c7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:46:40.885812 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.885789    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-kube-api-access-cq86l" pod "440dcea9-b109-41e6-aad5-bf7dcdaf4c7a" (UID: "440dcea9-b109-41e6-aad5-bf7dcdaf4c7a"). InnerVolumeSpecName "kube-api-access-cq86l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:40.985367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.985333    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:40.985367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.985367    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq86l\" (UniqueName: \"kubernetes.io/projected/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-kube-api-access-cq86l\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:40.985583 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:40.985379    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:46:41.560241 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:41.560205    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v" event={"ID":"440dcea9-b109-41e6-aad5-bf7dcdaf4c7a","Type":"ContainerDied","Data":"2dd71069e8ed05d62a5e40756aef4bfe19a8cd6f2ca5e71397ec9ad14c81d2f2"}
Nov 02 14:46:41.560713 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:41.560273    6284 scope.go:122] "RemoveContainer" containerID="e6c6f9cb63bb85fbac76f307c177abf993ce122e43f15e78fd3db36190bb508c"
Nov 02 14:46:41.560713 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:41.560221    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"
Nov 02 14:46:41.607647 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:41.607608    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"]
Nov 02 14:46:41.617438 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:41.617392    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fe1881-c885d4dbc-blk6v"]
Nov 02 14:46:42.170214 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:46:42.170180    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440dcea9-b109-41e6-aad5-bf7dcdaf4c7a" path="/var/lib/kubelet/pods/440dcea9-b109-41e6-aad5-bf7dcdaf4c7a/volumes"
Nov 02 14:46:43.896648 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:46:43.896606    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:47:00.084598 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:00.084550    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"]
Nov 02 14:47:13.896587 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:47:13.896543    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:47:15.136113 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.136062    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk" podUID="e5764355-3c40-4070-a6b1-54498b58ec46" containerName="main" containerID="cri-o://5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d" gracePeriod=45
Nov 02 14:47:15.437082 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.437048    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:47:15.517052 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.517016    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-tmp-dir\") pod \"e5764355-3c40-4070-a6b1-54498b58ec46\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") "
Nov 02 14:47:15.517290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.517107    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-home\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-home\") pod \"e5764355-3c40-4070-a6b1-54498b58ec46\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") "
Nov 02 14:47:15.517290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.517159    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-dshm\") pod \"e5764355-3c40-4070-a6b1-54498b58ec46\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") "
Nov 02 14:47:15.517290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.517257    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5764355-3c40-4070-a6b1-54498b58ec46-kube-api-access-gczwl\" (UniqueName: \"kubernetes.io/projected/e5764355-3c40-4070-a6b1-54498b58ec46-kube-api-access-gczwl\") pod \"e5764355-3c40-4070-a6b1-54498b58ec46\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") "
Nov 02 14:47:15.517510 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.517365    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5764355-3c40-4070-a6b1-54498b58ec46-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5764355-3c40-4070-a6b1-54498b58ec46-tls-certs\") pod \"e5764355-3c40-4070-a6b1-54498b58ec46\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") "
Nov 02 14:47:15.517510 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.517397    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-tmp-dir" pod "e5764355-3c40-4070-a6b1-54498b58ec46" (UID: "e5764355-3c40-4070-a6b1-54498b58ec46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:47:15.517510 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.517419    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-model-cache\") pod \"e5764355-3c40-4070-a6b1-54498b58ec46\" (UID: \"e5764355-3c40-4070-a6b1-54498b58ec46\") "
Nov 02 14:47:15.517684 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.517567    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-home" pod "e5764355-3c40-4070-a6b1-54498b58ec46" (UID: "e5764355-3c40-4070-a6b1-54498b58ec46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:47:15.517755 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.517728    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-model-cache" pod "e5764355-3c40-4070-a6b1-54498b58ec46" (UID: "e5764355-3c40-4070-a6b1-54498b58ec46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:47:15.518063 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.518040    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:15.518181 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.518068    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:15.518181 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.518081    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:15.519908 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.519877    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5764355-3c40-4070-a6b1-54498b58ec46-kube-api-access-gczwl" pod "e5764355-3c40-4070-a6b1-54498b58ec46" (UID: "e5764355-3c40-4070-a6b1-54498b58ec46"). InnerVolumeSpecName "kube-api-access-gczwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:15.520144 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.520121    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-dshm" pod "e5764355-3c40-4070-a6b1-54498b58ec46" (UID: "e5764355-3c40-4070-a6b1-54498b58ec46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:47:15.520144 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.520134    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5764355-3c40-4070-a6b1-54498b58ec46-tls-certs" pod "e5764355-3c40-4070-a6b1-54498b58ec46" (UID: "e5764355-3c40-4070-a6b1-54498b58ec46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:15.619462 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.619422    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5764355-3c40-4070-a6b1-54498b58ec46-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:15.619462 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.619456    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gczwl\" (UniqueName: \"kubernetes.io/projected/e5764355-3c40-4070-a6b1-54498b58ec46-kube-api-access-gczwl\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:15.619462 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.619465    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5764355-3c40-4070-a6b1-54498b58ec46-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:15.820400 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.820367    6284 generic.go:356] "Generic (PLEG): container finished" podID="e5764355-3c40-4070-a6b1-54498b58ec46" containerID="5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d" exitCode=0
Nov 02 14:47:15.820599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.820419    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"
Nov 02 14:47:15.820599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.820453    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk" event={"ID":"e5764355-3c40-4070-a6b1-54498b58ec46","Type":"ContainerDied","Data":"5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d"}
Nov 02 14:47:15.820599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.820510    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk" event={"ID":"e5764355-3c40-4070-a6b1-54498b58ec46","Type":"ContainerDied","Data":"936e91b2c0a018e72bc024d760080ace8dc1d6ba68ed814b96ddab1ea1e536f5"}
Nov 02 14:47:15.820599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.820528    6284 scope.go:122] "RemoveContainer" containerID="5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d"
Nov 02 14:47:15.832718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.832694    6284 scope.go:122] "RemoveContainer" containerID="5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d"
Nov 02 14:47:15.833153 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:47:15.833128    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d\": container with ID starting with 5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d not found: ID does not exist" containerID="5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d"
Nov 02 14:47:15.833244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.833166    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d"} err="failed to get container status \"5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d\": rpc error: code = NotFound desc = could not find container \"5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d\": container with ID starting with 5d0c6c72b36e1da25d5d9e93b330516153a44a8ff8963646f2985794a58e876d not found: ID does not exist"
Nov 02 14:47:15.868112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.868067    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"]
Nov 02 14:47:15.872226 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:15.872193    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-16c822/test-model-16c822-kserve-76fb78b79f-lvffk"]
Nov 02 14:47:16.175039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:16.172473    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5764355-3c40-4070-a6b1-54498b58ec46" path="/var/lib/kubelet/pods/e5764355-3c40-4070-a6b1-54498b58ec46/volumes"
Nov 02 14:47:43.296515 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.296481    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"]
Nov 02 14:47:43.296972 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.296699    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5" podUID="6e020ed5-2443-4833-b519-d6ddeb069b93" containerName="maas-api" containerID="cri-o://22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a" gracePeriod=30
Nov 02 14:47:43.677158 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.677123    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:47:43.730006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.728287    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-maas-api-tls\") pod \"6e020ed5-2443-4833-b519-d6ddeb069b93\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") "
Nov 02 14:47:43.730006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.728361    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e020ed5-2443-4833-b519-d6ddeb069b93-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e020ed5-2443-4833-b519-d6ddeb069b93-openshift-service-ca\") pod \"6e020ed5-2443-4833-b519-d6ddeb069b93\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") "
Nov 02 14:47:43.730006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.728407    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e020ed5-2443-4833-b519-d6ddeb069b93-kube-api-access-6w9mc\" (UniqueName: \"kubernetes.io/projected/6e020ed5-2443-4833-b519-d6ddeb069b93-kube-api-access-6w9mc\") pod \"6e020ed5-2443-4833-b519-d6ddeb069b93\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") "
Nov 02 14:47:43.730006 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.728889    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e020ed5-2443-4833-b519-d6ddeb069b93-openshift-service-ca" pod "6e020ed5-2443-4833-b519-d6ddeb069b93" (UID: "6e020ed5-2443-4833-b519-d6ddeb069b93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:47:43.732035 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.731997    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e020ed5-2443-4833-b519-d6ddeb069b93-kube-api-access-6w9mc" pod "6e020ed5-2443-4833-b519-d6ddeb069b93" (UID: "6e020ed5-2443-4833-b519-d6ddeb069b93"). InnerVolumeSpecName "kube-api-access-6w9mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:43.766067 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.765107    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-maas-api-tls" pod "6e020ed5-2443-4833-b519-d6ddeb069b93" (UID: "6e020ed5-2443-4833-b519-d6ddeb069b93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:43.829676 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.829637    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-metrics-certs\") pod \"6e020ed5-2443-4833-b519-d6ddeb069b93\" (UID: \"6e020ed5-2443-4833-b519-d6ddeb069b93\") "
Nov 02 14:47:43.830487 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.830456    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:43.830487 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.830490    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e020ed5-2443-4833-b519-d6ddeb069b93-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:43.830719 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.830507    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w9mc\" (UniqueName: \"kubernetes.io/projected/6e020ed5-2443-4833-b519-d6ddeb069b93-kube-api-access-6w9mc\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:43.832068 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.832034    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-metrics-certs" pod "6e020ed5-2443-4833-b519-d6ddeb069b93" (UID: "6e020ed5-2443-4833-b519-d6ddeb069b93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:43.902791 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:47:43.902701    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:47:43.931675 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:43.931637    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e020ed5-2443-4833-b519-d6ddeb069b93-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:44.070143 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:44.070099    6284 generic.go:356] "Generic (PLEG): container finished" podID="6e020ed5-2443-4833-b519-d6ddeb069b93" containerID="22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a" exitCode=0
Nov 02 14:47:44.070359 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:44.070173    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"
Nov 02 14:47:44.070359 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:44.070159    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5" event={"ID":"6e020ed5-2443-4833-b519-d6ddeb069b93","Type":"ContainerDied","Data":"22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a"}
Nov 02 14:47:44.070359 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:44.070312    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5" event={"ID":"6e020ed5-2443-4833-b519-d6ddeb069b93","Type":"ContainerDied","Data":"efaaab0e7dcf0b3f1da603dbe14412f9cc5b740c2863d2a46e9dab944ace7f80"}
Nov 02 14:47:44.070359 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:44.070344    6284 scope.go:122] "RemoveContainer" containerID="22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a"
Nov 02 14:47:44.082151 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:44.082121    6284 scope.go:122] "RemoveContainer" containerID="22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a"
Nov 02 14:47:44.082527 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:47:44.082504    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a\": container with ID starting with 22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a not found: ID does not exist" containerID="22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a"
Nov 02 14:47:44.082602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:44.082542    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a"} err="failed to get container status \"22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a\": rpc error: code = NotFound desc = could not find container \"22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a\": container with ID starting with 22f70cca5a4c317fca7be21d68851cbd8339d96f8767ebbb49a4a4773381044a not found: ID does not exist"
Nov 02 14:47:44.117426 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:44.117388    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"]
Nov 02 14:47:44.124010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:44.123974    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cbde78-ffb48d7bb-fg9j5"]
Nov 02 14:47:44.171094 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:44.171011    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e020ed5-2443-4833-b519-d6ddeb069b93" path="/var/lib/kubelet/pods/6e020ed5-2443-4833-b519-d6ddeb069b93/volumes"
Nov 02 14:47:47.219766 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.219726    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"]
Nov 02 14:47:47.220362 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.220054    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg" podUID="57fdfa91-1d6e-4cf5-876d-c586468c1e0e" containerName="maas-api" containerID="cri-o://a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674" gracePeriod=30
Nov 02 14:47:47.800778 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.800749    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:47:47.909908 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.909873    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-metrics-certs\") pod \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") "
Nov 02 14:47:47.910235 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.909998    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-openshift-service-ca\") pod \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") "
Nov 02 14:47:47.910235 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.910146    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-maas-api-tls\") pod \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") "
Nov 02 14:47:47.911001 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.910339    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-kube-api-access-nrv59\" (UniqueName: \"kubernetes.io/projected/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-kube-api-access-nrv59\") pod \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\" (UID: \"57fdfa91-1d6e-4cf5-876d-c586468c1e0e\") "
Nov 02 14:47:47.911001 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.910674    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-openshift-service-ca" pod "57fdfa91-1d6e-4cf5-876d-c586468c1e0e" (UID: "57fdfa91-1d6e-4cf5-876d-c586468c1e0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:47:47.911001 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.910902    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:47.913237 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.913078    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-maas-api-tls" pod "57fdfa91-1d6e-4cf5-876d-c586468c1e0e" (UID: "57fdfa91-1d6e-4cf5-876d-c586468c1e0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:47.913237 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.913138    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-metrics-certs" pod "57fdfa91-1d6e-4cf5-876d-c586468c1e0e" (UID: "57fdfa91-1d6e-4cf5-876d-c586468c1e0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:47.913237 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:47.913173    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-kube-api-access-nrv59" pod "57fdfa91-1d6e-4cf5-876d-c586468c1e0e" (UID: "57fdfa91-1d6e-4cf5-876d-c586468c1e0e"). InnerVolumeSpecName "kube-api-access-nrv59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:48.012074 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.012037    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrv59\" (UniqueName: \"kubernetes.io/projected/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-kube-api-access-nrv59\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:48.012074 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.012071    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:48.012074 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.012080    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57fdfa91-1d6e-4cf5-876d-c586468c1e0e-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:47:48.099746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.099652    6284 generic.go:356] "Generic (PLEG): container finished" podID="57fdfa91-1d6e-4cf5-876d-c586468c1e0e" containerID="a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674" exitCode=0
Nov 02 14:47:48.099746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.099704    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"
Nov 02 14:47:48.099746 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.099732    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg" event={"ID":"57fdfa91-1d6e-4cf5-876d-c586468c1e0e","Type":"ContainerDied","Data":"a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674"}
Nov 02 14:47:48.100033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.099791    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg" event={"ID":"57fdfa91-1d6e-4cf5-876d-c586468c1e0e","Type":"ContainerDied","Data":"21618be937c94b24d1fbe670c97dd5f69ff896eb9ce1df608a399ed6c97add25"}
Nov 02 14:47:48.100033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.099818    6284 scope.go:122] "RemoveContainer" containerID="a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674"
Nov 02 14:47:48.126911 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.126877    6284 scope.go:122] "RemoveContainer" containerID="a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674"
Nov 02 14:47:48.129168 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:47:48.129130    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674\": container with ID starting with a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674 not found: ID does not exist" containerID="a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674"
Nov 02 14:47:48.129398 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.129177    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674"} err="failed to get container status \"a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674\": rpc error: code = NotFound desc = could not find container \"a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674\": container with ID starting with a11997ebc5de07c6e0eb31a4e90c52ce970a45ae4df126e5a3dd84b10b08f674 not found: ID does not exist"
Nov 02 14:47:48.172065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.172027    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"]
Nov 02 14:47:48.172275 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:48.172128    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16c822-d6f97496d-mclsg"]
Nov 02 14:47:50.171131 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:50.171092    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57fdfa91-1d6e-4cf5-876d-c586468c1e0e" path="/var/lib/kubelet/pods/57fdfa91-1d6e-4cf5-876d-c586468c1e0e/volumes"
Nov 02 14:47:55.419495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.419447    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"]
Nov 02 14:47:55.420139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420123    6284 cpu_manager.go:414] "containerMap: removing container" podUID="d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" containerName="main"
Nov 02 14:47:55.420139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420140    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" containerName="main"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420155    6284 cpu_manager.go:414] "containerMap: removing container" podUID="0059df5c-839a-49af-8f54-2316292dbc49" containerName="maas-api"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420161    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0059df5c-839a-49af-8f54-2316292dbc49" containerName="maas-api"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420174    6284 cpu_manager.go:414] "containerMap: removing container" podUID="6e020ed5-2443-4833-b519-d6ddeb069b93" containerName="maas-api"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420182    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e020ed5-2443-4833-b519-d6ddeb069b93" containerName="maas-api"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420196    6284 cpu_manager.go:414] "containerMap: removing container" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420201    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420209    6284 cpu_manager.go:414] "containerMap: removing container" podUID="566c48d2-626a-469d-8dd5-25052bf31cf3" containerName="main"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420214    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="566c48d2-626a-469d-8dd5-25052bf31cf3" containerName="main"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420231    6284 cpu_manager.go:414] "containerMap: removing container" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerName="istio-proxy"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420235    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerName="istio-proxy"
Nov 02 14:47:55.420243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420244    6284 cpu_manager.go:414] "containerMap: removing container" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420249    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420258    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ad4dea4c-9cda-4aae-88be-a31096fcb38d" containerName="main"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420265    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4dea4c-9cda-4aae-88be-a31096fcb38d" containerName="main"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420276    6284 cpu_manager.go:414] "containerMap: removing container" podUID="4c1a8118-ecdb-48a7-af76-2c1fe30f71ec" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420281    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1a8118-ecdb-48a7-af76-2c1fe30f71ec" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420297    6284 cpu_manager.go:414] "containerMap: removing container" podUID="e5764355-3c40-4070-a6b1-54498b58ec46" containerName="main"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420301    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5764355-3c40-4070-a6b1-54498b58ec46" containerName="main"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420328    6284 cpu_manager.go:414] "containerMap: removing container" podUID="cf597423-a322-4d66-a75f-6fdda939de05" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420333    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf597423-a322-4d66-a75f-6fdda939de05" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420345    6284 cpu_manager.go:414] "containerMap: removing container" podUID="440dcea9-b109-41e6-aad5-bf7dcdaf4c7a" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420352    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440dcea9-b109-41e6-aad5-bf7dcdaf4c7a" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420362    6284 cpu_manager.go:414] "containerMap: removing container" podUID="57fdfa91-1d6e-4cf5-876d-c586468c1e0e" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420366    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fdfa91-1d6e-4cf5-876d-c586468c1e0e" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420374    6284 cpu_manager.go:414] "containerMap: removing container" podUID="106517f1-b322-462b-80ba-ae2c8316bc62" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420379    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106517f1-b322-462b-80ba-ae2c8316bc62" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420386    6284 cpu_manager.go:414] "containerMap: removing container" podUID="05adda46-8560-440c-a0f6-8ae9252bb27a" containerName="main"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420391    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05adda46-8560-440c-a0f6-8ae9252bb27a" containerName="main"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420400    6284 cpu_manager.go:414] "containerMap: removing container" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420406    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420508    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5764355-3c40-4070-a6b1-54498b58ec46" containerName="main"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420519    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf597423-a322-4d66-a75f-6fdda939de05" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420526    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="440dcea9-b109-41e6-aad5-bf7dcdaf4c7a" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420534    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fdfa91-1d6e-4cf5-876d-c586468c1e0e" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420541    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="106517f1-b322-462b-80ba-ae2c8316bc62" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420547    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="05adda46-8560-440c-a0f6-8ae9252bb27a" containerName="main"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420554    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68a59a0-b4cf-40e0-9f58-4f0db1436596" containerName="istio-proxy"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420560    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5bbcbf6-e1fa-40c5-a873-8f5e2e27beb9" containerName="main"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420565    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="0059df5c-839a-49af-8f54-2316292dbc49" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420570    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e020ed5-2443-4833-b519-d6ddeb069b93" containerName="maas-api"
Nov 02 14:47:55.420571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420577    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="94cc467e-61a3-4f4b-b317-60fe7deffb63" containerName="istio-proxy"
Nov 02 14:47:55.421553 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420587    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="566c48d2-626a-469d-8dd5-25052bf31cf3" containerName="main"
Nov 02 14:47:55.421553 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420597    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ca6186-3243-459b-ad2a-7d6584ddf0d2" containerName="istio-proxy"
Nov 02 14:47:55.421553 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420603    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="56655b89-7919-4694-8933-fbcc61b81de0" containerName="istio-proxy"
Nov 02 14:47:55.421553 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420609    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4dea4c-9cda-4aae-88be-a31096fcb38d" containerName="main"
Nov 02 14:47:55.421553 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.420616    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1a8118-ecdb-48a7-af76-2c1fe30f71ec" containerName="maas-api"
Nov 02 14:47:55.426073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.426030    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.458305 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.458254    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"]
Nov 02 14:47:55.520691 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.520644    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hztnk\" (UniqueName: \"kubernetes.io/projected/578e528e-aff8-416c-94b5-9bf6cc01a4cc-kube-api-access-hztnk\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.520886 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.520701    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.520886 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.520825    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.520886 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.520859    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.521055 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.520930    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.521055 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.521014    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.521122 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.521108    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.521178 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.521163    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.521238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.521226    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.622340 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.622287    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.622546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.622393    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.622546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.622417    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.622546 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.622500    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hztnk\" (UniqueName: \"kubernetes.io/projected/578e528e-aff8-416c-94b5-9bf6cc01a4cc-kube-api-access-hztnk\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.622728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.622545    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.622728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.622618    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.622728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.622642    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.622881 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.622740    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.622881 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.622793    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.623145 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.623117    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.623271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.623164    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.623325 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.623287    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.623390 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.623311    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.623646 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.623620    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.625395 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.625366    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.625684 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.625661    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.645654 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.645623    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/578e528e-aff8-416c-94b5-9bf6cc01a4cc-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.645860 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.645837    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hztnk\" (UniqueName: \"kubernetes.io/projected/578e528e-aff8-416c-94b5-9bf6cc01a4cc-kube-api-access-hztnk\") pod \"maas-default-gateway-openshift-default-6886b96f9-dpxqg\" (UID: \"578e528e-aff8-416c-94b5-9bf6cc01a4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:55.749437 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:55.749390    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:56.084276 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:56.084241    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"]
Nov 02 14:47:56.086994 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:47:56.086961    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578e528e_aff8_416c_94b5_9bf6cc01a4cc.slice/crio-a45eeff297d863eaa5f1a6f024cafc6a57f07486b5a970d55a8f9d4ee7685104 WatchSource:0}: Error finding container a45eeff297d863eaa5f1a6f024cafc6a57f07486b5a970d55a8f9d4ee7685104: Status 404 returned error can't find the container with id a45eeff297d863eaa5f1a6f024cafc6a57f07486b5a970d55a8f9d4ee7685104
Nov 02 14:47:56.090064 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:56.090019    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:47:56.090182 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:56.090113    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:47:56.090182 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:56.090172    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:47:56.171387 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:56.171345    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" event={"ID":"578e528e-aff8-416c-94b5-9bf6cc01a4cc","Type":"ContainerStarted","Data":"a45eeff297d863eaa5f1a6f024cafc6a57f07486b5a970d55a8f9d4ee7685104"}
Nov 02 14:47:57.207539 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:57.207491    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" event={"ID":"578e528e-aff8-416c-94b5-9bf6cc01a4cc","Type":"ContainerStarted","Data":"8b926e24dd563dc01f6fb110db3c58596fb9dea60b4c8dd128f0682d3d743ba3"}
Nov 02 14:47:57.272605 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:57.271668    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" podStartSLOduration=2.271647629 podStartE2EDuration="2.271647629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:47:57.270500842 +0000 UTC m=+3809.229876450" watchObservedRunningTime="2026-11-02 14:47:57.271647629 +0000 UTC m=+3809.231023237"
Nov 02 14:47:57.763028 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:57.752993    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:47:57.774551 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:57.774492    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dpxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:47:57.774756 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:57.774568    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" podUID="578e528e-aff8-416c-94b5-9bf6cc01a4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 14:47:58.767962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:58.754427    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dpxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:47:58.767962 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:58.754496    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" podUID="578e528e-aff8-416c-94b5-9bf6cc01a4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 14:47:59.769960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:59.769454    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dpxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:47:59.769960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:47:59.769573    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" podUID="578e528e-aff8-416c-94b5-9bf6cc01a4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 14:48:00.765961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:00.759505    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dpxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:48:00.765961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:00.759581    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" podUID="578e528e-aff8-416c-94b5-9bf6cc01a4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 14:48:01.771414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:01.771372    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dpxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:48:01.772092 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:01.772059    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" podUID="578e528e-aff8-416c-94b5-9bf6cc01a4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 14:48:02.762963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:02.752497    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dpxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:48:02.762963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:02.752584    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" podUID="578e528e-aff8-416c-94b5-9bf6cc01a4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 14:48:03.750723 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:03.750677    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dpxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:48:03.751161 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:03.750755    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" podUID="578e528e-aff8-416c-94b5-9bf6cc01a4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 02 14:48:05.282223 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:05.282154    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:48:05.282843 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:05.282819    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:48:05.284706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:05.284683    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg"
Nov 02 14:48:10.455286 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.455245    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"]
Nov 02 14:48:10.461340 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.461302    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.466712 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.466677    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.466947 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.466750    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e32694-fbfc-4adc-aef1-7936996fe885-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.466947 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.466796    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8tzh\" (UniqueName: \"kubernetes.io/projected/17e32694-fbfc-4adc-aef1-7936996fe885-kube-api-access-m8tzh\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.466947 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.466823    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.466947 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.466915    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e32694-fbfc-4adc-aef1-7936996fe885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.467171 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.467055    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.467171 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.467146    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.467278 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.467249    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.467334 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.467279    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e32694-fbfc-4adc-aef1-7936996fe885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.487199 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.487139    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"]
Nov 02 14:48:10.567947 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.567891    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568173 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.567983    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e32694-fbfc-4adc-aef1-7936996fe885-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568173 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568026    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8tzh\" (UniqueName: \"kubernetes.io/projected/17e32694-fbfc-4adc-aef1-7936996fe885-kube-api-access-m8tzh\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568173 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568145    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568363 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568193    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e32694-fbfc-4adc-aef1-7936996fe885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568363 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568239    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568363 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568286    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568388    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568401    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568430    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e32694-fbfc-4adc-aef1-7936996fe885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568682 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568520    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.568911 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568886    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.569065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.568916    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.569224 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.569203    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e32694-fbfc-4adc-aef1-7936996fe885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.570679 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.570658    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e32694-fbfc-4adc-aef1-7936996fe885-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.571052 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.571034    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e32694-fbfc-4adc-aef1-7936996fe885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.589732 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.589701    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e32694-fbfc-4adc-aef1-7936996fe885-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.589929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.589846    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8tzh\" (UniqueName: \"kubernetes.io/projected/17e32694-fbfc-4adc-aef1-7936996fe885-kube-api-access-m8tzh\") pod \"maas-default-gateway-openshift-default-6886b96f9-lfj9p\" (UID: \"17e32694-fbfc-4adc-aef1-7936996fe885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:10.784880 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:10.784835    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:11.122812 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:11.122777    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"]
Nov 02 14:48:11.124735 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:48:11.124707    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e32694_fbfc_4adc_aef1_7936996fe885.slice/crio-c4b0dafe9fa124b94a30bd8b649eaf84f17145f3c3911b253f13cc829347cb9c WatchSource:0}: Error finding container c4b0dafe9fa124b94a30bd8b649eaf84f17145f3c3911b253f13cc829347cb9c: Status 404 returned error can't find the container with id c4b0dafe9fa124b94a30bd8b649eaf84f17145f3c3911b253f13cc829347cb9c
Nov 02 14:48:11.127631 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:11.127598    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:48:11.127709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:11.127672    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:48:11.127709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:11.127703    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:48:11.350611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:11.350567    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" event={"ID":"17e32694-fbfc-4adc-aef1-7936996fe885","Type":"ContainerStarted","Data":"1da396831ef03a96ed18a0829d5f01c97ce25adcb83c6ac47929798d88bd3eea"}
Nov 02 14:48:11.350887 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:11.350675    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" event={"ID":"17e32694-fbfc-4adc-aef1-7936996fe885","Type":"ContainerStarted","Data":"c4b0dafe9fa124b94a30bd8b649eaf84f17145f3c3911b253f13cc829347cb9c"}
Nov 02 14:48:11.402696 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:11.402572    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" podStartSLOduration=1.402551011 podStartE2EDuration="1.402551011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:48:11.398509349 +0000 UTC m=+3823.357884968" watchObservedRunningTime="2026-11-02 14:48:11.402551011 +0000 UTC m=+3823.361926617"
Nov 02 14:48:11.785003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:11.784969    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:12.787451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:12.787412    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lfj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 14:48:12.787973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:12.787498    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" podUID="17e32694-fbfc-4adc-aef1-7936996fe885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 14:48:13.785679 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:13.785640    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lfj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 14:48:13.785877 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:13.785701    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" podUID="17e32694-fbfc-4adc-aef1-7936996fe885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 14:48:13.906192 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:48:13.906149    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:48:14.801144 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:14.800498    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lfj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 14:48:14.801144 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:14.800563    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" podUID="17e32694-fbfc-4adc-aef1-7936996fe885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 14:48:15.785682 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:15.785641    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lfj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 14:48:15.786126 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:15.785699    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" podUID="17e32694-fbfc-4adc-aef1-7936996fe885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 14:48:16.785374 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:16.785337    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lfj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 14:48:16.785577 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:16.785394    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" podUID="17e32694-fbfc-4adc-aef1-7936996fe885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 14:48:17.786096 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:17.786056    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lfj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 14:48:17.786574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:17.786135    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" podUID="17e32694-fbfc-4adc-aef1-7936996fe885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 14:48:18.785542 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:18.785499    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lfj9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 14:48:18.785904 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:18.785559    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" podUID="17e32694-fbfc-4adc-aef1-7936996fe885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 14:48:20.094848 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:20.094808    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:20.095348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:20.095268    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:20.096066 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:20.096045    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p"
Nov 02 14:48:43.897513 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:48:43.897466    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:48:44.939639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:44.939593    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"]
Nov 02 14:48:44.942250 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:44.940280    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82" podUID="afd5fc01-bcca-445f-8779-561d8aea2504" containerName="maas-api" containerID="cri-o://3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f" gracePeriod=30
Nov 02 14:48:45.330037 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.330006    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:48:45.439865 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.439826    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd5fc01-bcca-445f-8779-561d8aea2504-kube-api-access-nwqbt\" (UniqueName: \"kubernetes.io/projected/afd5fc01-bcca-445f-8779-561d8aea2504-kube-api-access-nwqbt\") pod \"afd5fc01-bcca-445f-8779-561d8aea2504\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") "
Nov 02 14:48:45.440280 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.440251    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-maas-api-tls\") pod \"afd5fc01-bcca-445f-8779-561d8aea2504\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") "
Nov 02 14:48:45.440465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.440448    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd5fc01-bcca-445f-8779-561d8aea2504-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd5fc01-bcca-445f-8779-561d8aea2504-openshift-service-ca\") pod \"afd5fc01-bcca-445f-8779-561d8aea2504\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") "
Nov 02 14:48:45.440714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.440695    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-metrics-certs\") pod \"afd5fc01-bcca-445f-8779-561d8aea2504\" (UID: \"afd5fc01-bcca-445f-8779-561d8aea2504\") "
Nov 02 14:48:45.444332 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.441638    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd5fc01-bcca-445f-8779-561d8aea2504-openshift-service-ca" pod "afd5fc01-bcca-445f-8779-561d8aea2504" (UID: "afd5fc01-bcca-445f-8779-561d8aea2504"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:48:45.445289 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.445256    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd5fc01-bcca-445f-8779-561d8aea2504-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:48:45.445743 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.445702    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-maas-api-tls" pod "afd5fc01-bcca-445f-8779-561d8aea2504" (UID: "afd5fc01-bcca-445f-8779-561d8aea2504"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:48:45.446158 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.446122    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd5fc01-bcca-445f-8779-561d8aea2504-kube-api-access-nwqbt" pod "afd5fc01-bcca-445f-8779-561d8aea2504" (UID: "afd5fc01-bcca-445f-8779-561d8aea2504"). InnerVolumeSpecName "kube-api-access-nwqbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:45.448584 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.448550    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-metrics-certs" pod "afd5fc01-bcca-445f-8779-561d8aea2504" (UID: "afd5fc01-bcca-445f-8779-561d8aea2504"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:48:45.546742 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.546645    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:48:45.546742 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.546681    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd5fc01-bcca-445f-8779-561d8aea2504-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:48:45.546742 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.546691    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwqbt\" (UniqueName: \"kubernetes.io/projected/afd5fc01-bcca-445f-8779-561d8aea2504-kube-api-access-nwqbt\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:48:45.626048 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.626009    6284 generic.go:356] "Generic (PLEG): container finished" podID="afd5fc01-bcca-445f-8779-561d8aea2504" containerID="3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f" exitCode=0
Nov 02 14:48:45.626238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.626064    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"
Nov 02 14:48:45.626238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.626063    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82" event={"ID":"afd5fc01-bcca-445f-8779-561d8aea2504","Type":"ContainerDied","Data":"3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f"}
Nov 02 14:48:45.626238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.626187    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82" event={"ID":"afd5fc01-bcca-445f-8779-561d8aea2504","Type":"ContainerDied","Data":"d890162de7b5ebe32cd18a8ea48c090d8c3b1733a636ace709164c49a941a5bb"}
Nov 02 14:48:45.626238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.626209    6284 scope.go:122] "RemoveContainer" containerID="3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f"
Nov 02 14:48:45.636606 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.636588    6284 scope.go:122] "RemoveContainer" containerID="3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f"
Nov 02 14:48:45.636884 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:48:45.636859    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f\": container with ID starting with 3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f not found: ID does not exist" containerID="3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f"
Nov 02 14:48:45.637021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.636892    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f"} err="failed to get container status \"3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f\": rpc error: code = NotFound desc = could not find container \"3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f\": container with ID starting with 3ceda0cbb9b4eab20bc76fafb1e5450814adf28adbee54b668d2fddeeeea2e3f not found: ID does not exist"
Nov 02 14:48:45.672122 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.672082    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"]
Nov 02 14:48:45.681527 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:45.681492    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82"]
Nov 02 14:48:46.173280 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:46.173241    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd5fc01-bcca-445f-8779-561d8aea2504" path="/var/lib/kubelet/pods/afd5fc01-bcca-445f-8779-561d8aea2504/volumes"
Nov 02 14:48:48.271634 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:48:48.271581    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea3e8f-84cfdcb5ff-vwc82" podUID="afd5fc01-bcca-445f-8779-561d8aea2504" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.158:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:49:13.897409 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:49:13.897364    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:49:16.894869 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:16.894831    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"]
Nov 02 14:49:16.895569 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:16.895086    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5" podUID="de777481-cbf8-4999-a4c6-d1b2687d2afe" containerName="maas-api" containerID="cri-o://cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d" gracePeriod=30
Nov 02 14:49:17.300150 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.300114    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:49:17.342977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.338047    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-metrics-certs\") pod \"de777481-cbf8-4999-a4c6-d1b2687d2afe\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") "
Nov 02 14:49:17.342977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.338200    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de777481-cbf8-4999-a4c6-d1b2687d2afe-kube-api-access-fj24k\" (UniqueName: \"kubernetes.io/projected/de777481-cbf8-4999-a4c6-d1b2687d2afe-kube-api-access-fj24k\") pod \"de777481-cbf8-4999-a4c6-d1b2687d2afe\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") "
Nov 02 14:49:17.342977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.338424    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de777481-cbf8-4999-a4c6-d1b2687d2afe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de777481-cbf8-4999-a4c6-d1b2687d2afe-openshift-service-ca\") pod \"de777481-cbf8-4999-a4c6-d1b2687d2afe\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") "
Nov 02 14:49:17.342977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.338597    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-maas-api-tls\") pod \"de777481-cbf8-4999-a4c6-d1b2687d2afe\" (UID: \"de777481-cbf8-4999-a4c6-d1b2687d2afe\") "
Nov 02 14:49:17.386961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.368736    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de777481-cbf8-4999-a4c6-d1b2687d2afe-openshift-service-ca" pod "de777481-cbf8-4999-a4c6-d1b2687d2afe" (UID: "de777481-cbf8-4999-a4c6-d1b2687d2afe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:49:17.391959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.388489    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-maas-api-tls" pod "de777481-cbf8-4999-a4c6-d1b2687d2afe" (UID: "de777481-cbf8-4999-a4c6-d1b2687d2afe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:49:17.392750 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.392709    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-metrics-certs" pod "de777481-cbf8-4999-a4c6-d1b2687d2afe" (UID: "de777481-cbf8-4999-a4c6-d1b2687d2afe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:49:17.407181 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.407139    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de777481-cbf8-4999-a4c6-d1b2687d2afe-kube-api-access-fj24k" pod "de777481-cbf8-4999-a4c6-d1b2687d2afe" (UID: "de777481-cbf8-4999-a4c6-d1b2687d2afe"). InnerVolumeSpecName "kube-api-access-fj24k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:49:17.440626 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.440500    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:49:17.440626 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.440549    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de777481-cbf8-4999-a4c6-d1b2687d2afe-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:49:17.440626 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.440564    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj24k\" (UniqueName: \"kubernetes.io/projected/de777481-cbf8-4999-a4c6-d1b2687d2afe-kube-api-access-fj24k\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:49:17.440626 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.440582    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de777481-cbf8-4999-a4c6-d1b2687d2afe-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:49:17.861239 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.861199    6284 generic.go:356] "Generic (PLEG): container finished" podID="de777481-cbf8-4999-a4c6-d1b2687d2afe" containerID="cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d" exitCode=0
Nov 02 14:49:17.861702 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.861673    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5" event={"ID":"de777481-cbf8-4999-a4c6-d1b2687d2afe","Type":"ContainerDied","Data":"cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d"}
Nov 02 14:49:17.861883 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.861867    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5" event={"ID":"de777481-cbf8-4999-a4c6-d1b2687d2afe","Type":"ContainerDied","Data":"3a5eec5f334c8ab9f481191bd7e576371bd16cd03e67c3ad9b0954fac085e96f"}
Nov 02 14:49:17.862143 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.862117    6284 scope.go:122] "RemoveContainer" containerID="cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d"
Nov 02 14:49:17.862618 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.862593    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"
Nov 02 14:49:17.878649 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.878625    6284 scope.go:122] "RemoveContainer" containerID="cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d"
Nov 02 14:49:17.879072 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:49:17.879044    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d\": container with ID starting with cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d not found: ID does not exist" containerID="cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d"
Nov 02 14:49:17.879163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.879082    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d"} err="failed to get container status \"cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d\": rpc error: code = NotFound desc = could not find container \"cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d\": container with ID starting with cbaf2a995b3fa25b0f5f154c23a5ec0e61e6b2109bf5dd5a1c6f9d9a223e260d not found: ID does not exist"
Nov 02 14:49:17.913632 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.913595    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"]
Nov 02 14:49:17.917264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:17.917233    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c8f647-5b88f59478-48cd5"]
Nov 02 14:49:18.180190 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:18.180097    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de777481-cbf8-4999-a4c6-d1b2687d2afe" path="/var/lib/kubelet/pods/de777481-cbf8-4999-a4c6-d1b2687d2afe/volumes"
Nov 02 14:49:29.216501 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:29.216472    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:49:29.228555 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:49:29.228521    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:49:43.896020 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:49:43.895974    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:50:13.896166 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:50:13.896117    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:50:14.736764 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:14.736723    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"]
Nov 02 14:50:14.737021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:14.736990    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7" podUID="e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" containerName="maas-api" containerID="cri-o://d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827" gracePeriod=30
Nov 02 14:50:14.821726 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:14.821666    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7" podUID="e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.160:8443/health\": dial tcp 10.134.0.160:8443: connect: connection refused"
Nov 02 14:50:15.095219 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.094752    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:50:15.289733 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.289633    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-metrics-certs\") pod \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") "
Nov 02 14:50:15.289965 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.289770    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-maas-api-tls\") pod \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") "
Nov 02 14:50:15.289965 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.289811    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-openshift-service-ca\") pod \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") "
Nov 02 14:50:15.289965 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.289835    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-kube-api-access-jw4pc\" (UniqueName: \"kubernetes.io/projected/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-kube-api-access-jw4pc\") pod \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\" (UID: \"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9\") "
Nov 02 14:50:15.290254 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.290218    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-openshift-service-ca" pod "e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" (UID: "e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:50:15.290650 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.290624    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:50:15.292296 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.292269    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-metrics-certs" pod "e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" (UID: "e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:50:15.292491 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.292475    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-kube-api-access-jw4pc" pod "e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" (UID: "e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9"). InnerVolumeSpecName "kube-api-access-jw4pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:50:15.292621 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.292600    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-maas-api-tls" pod "e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" (UID: "e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:50:15.326995 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.326959    6284 generic.go:356] "Generic (PLEG): container finished" podID="e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" containerID="d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827" exitCode=0
Nov 02 14:50:15.327184 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.326999    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7" event={"ID":"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9","Type":"ContainerDied","Data":"d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827"}
Nov 02 14:50:15.327184 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.327022    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"
Nov 02 14:50:15.327184 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.327079    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7" event={"ID":"e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9","Type":"ContainerDied","Data":"fd4526f990956920a390ead70d5c083201c751e42ef718abcb9bea1b7231740c"}
Nov 02 14:50:15.327184 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.327113    6284 scope.go:122] "RemoveContainer" containerID="d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827"
Nov 02 14:50:15.343469 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.342719    6284 scope.go:122] "RemoveContainer" containerID="d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827"
Nov 02 14:50:15.343639 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:50:15.343496    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827\": container with ID starting with d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827 not found: ID does not exist" containerID="d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827"
Nov 02 14:50:15.343639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.343544    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827"} err="failed to get container status \"d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827\": rpc error: code = NotFound desc = could not find container \"d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827\": container with ID starting with d6e72c32fb85f043296afd3f8b2ff9d8b3131e708c041042749da8c7cc3cd827 not found: ID does not exist"
Nov 02 14:50:15.381240 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.381199    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"]
Nov 02 14:50:15.384422 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.384384    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-80133f-7b778d9df4-2p4f7"]
Nov 02 14:50:15.391687 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.391641    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:50:15.391687 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.391684    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:50:15.391919 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:15.391701    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw4pc\" (UniqueName: \"kubernetes.io/projected/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9-kube-api-access-jw4pc\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:50:16.170686 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:16.170647    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" path="/var/lib/kubelet/pods/e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9/volumes"
Nov 02 14:50:22.198041 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.197918    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"]
Nov 02 14:50:22.198632 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.198610    6284 cpu_manager.go:414] "containerMap: removing container" podUID="e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" containerName="maas-api"
Nov 02 14:50:22.198632 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.198630    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" containerName="maas-api"
Nov 02 14:50:22.198814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.198653    6284 cpu_manager.go:414] "containerMap: removing container" podUID="de777481-cbf8-4999-a4c6-d1b2687d2afe" containerName="maas-api"
Nov 02 14:50:22.198814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.198659    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de777481-cbf8-4999-a4c6-d1b2687d2afe" containerName="maas-api"
Nov 02 14:50:22.198814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.198682    6284 cpu_manager.go:414] "containerMap: removing container" podUID="afd5fc01-bcca-445f-8779-561d8aea2504" containerName="maas-api"
Nov 02 14:50:22.198814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.198687    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd5fc01-bcca-445f-8779-561d8aea2504" containerName="maas-api"
Nov 02 14:50:22.198814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.198764    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="de777481-cbf8-4999-a4c6-d1b2687d2afe" containerName="maas-api"
Nov 02 14:50:22.198814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.198775    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd5fc01-bcca-445f-8779-561d8aea2504" containerName="maas-api"
Nov 02 14:50:22.198814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.198786    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c1fd3b-8123-4a2a-acc9-c5a45b46d3d9" containerName="maas-api"
Nov 02 14:50:22.202356 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.202331    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.227093 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.227044    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-fcefcc\"/\"openshift-service-ca.crt\""
Nov 02 14:50:22.227093 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.227084    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-fcefcc\"/\"kube-root-ca.crt\""
Nov 02 14:50:22.227292 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.227107    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-fcefcc\"/\"default-dockercfg-q7f7j\""
Nov 02 14:50:22.227351 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.227322    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-fcefcc\"/\"sub-test-model-fcefcc-kserve-self-signed-certs\""
Nov 02 14:50:22.232078 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.232041    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"]
Nov 02 14:50:22.287088 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.287050    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-home\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.287656 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.287366    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tls-certs\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.287806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.287657    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tmp-dir\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.287806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.287732    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnn7q\" (UniqueName: \"kubernetes.io/projected/0eb75de2-3838-4b3e-a5ab-85e2426628e7-kube-api-access-jnn7q\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.287965 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.287822    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-dshm\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.287965 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.287847    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-model-cache\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.389636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.389566    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tmp-dir\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.389912 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.389879    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnn7q\" (UniqueName: \"kubernetes.io/projected/0eb75de2-3838-4b3e-a5ab-85e2426628e7-kube-api-access-jnn7q\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.390083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.389949    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-dshm\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.390083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.389973    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-model-cache\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.390083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.390042    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-home\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.390249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.390076    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tmp-dir\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.390249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.390225    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tls-certs\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.390494 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.390467    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-home\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.390694 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.390670    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-model-cache\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.392520 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.392494    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-dshm\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.393169 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.393147    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tls-certs\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.411501 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.411463    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnn7q\" (UniqueName: \"kubernetes.io/projected/0eb75de2-3838-4b3e-a5ab-85e2426628e7-kube-api-access-jnn7q\") pod \"sub-test-model-fcefcc-kserve-6b68557b85-gcf2c\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.547921 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.547879    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:22.895515 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:22.895436    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"]
Nov 02 14:50:22.898421 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:50:22.898371    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0eb75de2_3838_4b3e_a5ab_85e2426628e7.slice/crio-3363b86251c6f51211e70fdae4bd4277e7b9f368493e4945260f1378c41155b9 WatchSource:0}: Error finding container 3363b86251c6f51211e70fdae4bd4277e7b9f368493e4945260f1378c41155b9: Status 404 returned error can't find the container with id 3363b86251c6f51211e70fdae4bd4277e7b9f368493e4945260f1378c41155b9
Nov 02 14:50:23.388725 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:23.388688    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c" event={"ID":"0eb75de2-3838-4b3e-a5ab-85e2426628e7","Type":"ContainerStarted","Data":"d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea"}
Nov 02 14:50:23.389192 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:23.388734    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c" event={"ID":"0eb75de2-3838-4b3e-a5ab-85e2426628e7","Type":"ContainerStarted","Data":"3363b86251c6f51211e70fdae4bd4277e7b9f368493e4945260f1378c41155b9"}
Nov 02 14:50:23.428399 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:23.428326    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c" podStartSLOduration=1.428302631 podStartE2EDuration="1.428302631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:50:23.425044461 +0000 UTC m=+3955.384420071" watchObservedRunningTime="2026-11-02 14:50:23.428302631 +0000 UTC m=+3955.387678240"
Nov 02 14:50:23.549107 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:23.549054    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:32.551133 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:32.551088    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:32.566333 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:32.566295    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:33.477167 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:50:33.477132    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:50:43.896583 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:50:43.896538    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:51:13.898447 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:51:13.898403    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:51:15.096636 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.096597    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"]
Nov 02 14:51:15.097465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.097409    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2" podUID="daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50" containerName="maas-api" containerID="cri-o://4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6" gracePeriod=30
Nov 02 14:51:15.464841 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.464804    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:51:15.495969 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.495508    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-kube-api-access-vq6dp\" (UniqueName: \"kubernetes.io/projected/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-kube-api-access-vq6dp\") pod \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") "
Nov 02 14:51:15.495969 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.495651    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-openshift-service-ca\") pod \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") "
Nov 02 14:51:15.495969 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.495726    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-maas-api-tls\") pod \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") "
Nov 02 14:51:15.495969 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.495825    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-metrics-certs\") pod \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\" (UID: \"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50\") "
Nov 02 14:51:15.513960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.511224    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-openshift-service-ca" pod "daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50" (UID: "daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:51:15.524309 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.515425    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-maas-api-tls" pod "daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50" (UID: "daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:51:15.535963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.527476    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-kube-api-access-vq6dp" pod "daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50" (UID: "daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50"). InnerVolumeSpecName "kube-api-access-vq6dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:51:15.535963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.533265    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-metrics-certs" pod "daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50" (UID: "daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:51:15.602957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.599053    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:15.602957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.599096    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:15.602957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.599113    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6dp\" (UniqueName: \"kubernetes.io/projected/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-kube-api-access-vq6dp\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:15.602957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.599126    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:15.780766 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.780729    6284 generic.go:356] "Generic (PLEG): container finished" podID="daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50" containerID="4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6" exitCode=0
Nov 02 14:51:15.781003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.780782    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"
Nov 02 14:51:15.781003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.780812    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2" event={"ID":"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50","Type":"ContainerDied","Data":"4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6"}
Nov 02 14:51:15.781003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.780904    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2" event={"ID":"daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50","Type":"ContainerDied","Data":"44afccf4ad9b6d07c6e8440a91829d5412c85c465fe03dc204306c34c9aaf95f"}
Nov 02 14:51:15.781003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.780928    6284 scope.go:122] "RemoveContainer" containerID="4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6"
Nov 02 14:51:15.792396 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.792368    6284 scope.go:122] "RemoveContainer" containerID="4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6"
Nov 02 14:51:15.792752 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:51:15.792731    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6\": container with ID starting with 4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6 not found: ID does not exist" containerID="4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6"
Nov 02 14:51:15.792817 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.792764    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6"} err="failed to get container status \"4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6\": rpc error: code = NotFound desc = could not find container \"4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6\": container with ID starting with 4acd275811c5234ca374eb2e7228e2c2846949b942ae4d1141170c217401dac6 not found: ID does not exist"
Nov 02 14:51:15.834628 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.834586    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"]
Nov 02 14:51:15.839653 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:15.839619    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-87efd3-5675b6b87-zsgb2"]
Nov 02 14:51:16.170944 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:16.170880    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50" path="/var/lib/kubelet/pods/daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50/volumes"
Nov 02 14:51:28.446847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:28.446808    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"]
Nov 02 14:51:33.447478 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:33.447422    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g" podUID="9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4" containerName="payload-processing" containerID="cri-o://69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb" gracePeriod=25
Nov 02 14:51:36.488680 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:36.488636    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"]
Nov 02 14:51:36.490742 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:36.490699    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w" podUID="869b27f2-7068-4501-8d4b-4aae0e159f5e" containerName="discovery" containerID="cri-o://1490046076bcb7cb42f6588d0deff053a9beca33c4fe23e5c7d50582a20dff02" gracePeriod=30
Nov 02 14:51:37.032104 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.030789    6284 generic.go:356] "Generic (PLEG): container finished" podID="869b27f2-7068-4501-8d4b-4aae0e159f5e" containerID="1490046076bcb7cb42f6588d0deff053a9beca33c4fe23e5c7d50582a20dff02" exitCode=0
Nov 02 14:51:37.032104 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.030853    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w" event={"ID":"869b27f2-7068-4501-8d4b-4aae0e159f5e","Type":"ContainerDied","Data":"1490046076bcb7cb42f6588d0deff053a9beca33c4fe23e5c7d50582a20dff02"}
Nov 02 14:51:37.047111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.047083    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:51:37.158963 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.148830    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-ca-configmap\") pod \"869b27f2-7068-4501-8d4b-4aae0e159f5e\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") "
Nov 02 14:51:37.167961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.155032    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-ca-configmap" pod "869b27f2-7068-4501-8d4b-4aae0e159f5e" (UID: "869b27f2-7068-4501-8d4b-4aae0e159f5e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:51:37.167961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.161139    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/869b27f2-7068-4501-8d4b-4aae0e159f5e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/869b27f2-7068-4501-8d4b-4aae0e159f5e-local-certs\") pod \"869b27f2-7068-4501-8d4b-4aae0e159f5e\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") "
Nov 02 14:51:37.167961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.164756    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869b27f2-7068-4501-8d4b-4aae0e159f5e-local-certs" pod "869b27f2-7068-4501-8d4b-4aae0e159f5e" (UID: "869b27f2-7068-4501-8d4b-4aae0e159f5e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:51:37.181181 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.180826    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-dns-cert" pod "869b27f2-7068-4501-8d4b-4aae0e159f5e" (UID: "869b27f2-7068-4501-8d4b-4aae0e159f5e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:51:37.184247 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.182435    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-dns-cert\") pod \"869b27f2-7068-4501-8d4b-4aae0e159f5e\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") "
Nov 02 14:51:37.184247 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.182507    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-kubeconfig\") pod \"869b27f2-7068-4501-8d4b-4aae0e159f5e\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") "
Nov 02 14:51:37.184247 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.182548    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-kube-api-access-x2njk\" (UniqueName: \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-kube-api-access-x2njk\") pod \"869b27f2-7068-4501-8d4b-4aae0e159f5e\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") "
Nov 02 14:51:37.184247 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.182601    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-cacerts\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-cacerts\") pod \"869b27f2-7068-4501-8d4b-4aae0e159f5e\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") "
Nov 02 14:51:37.184247 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.182747    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-token\" (UniqueName: \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-token\") pod \"869b27f2-7068-4501-8d4b-4aae0e159f5e\" (UID: \"869b27f2-7068-4501-8d4b-4aae0e159f5e\") "
Nov 02 14:51:37.184247 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.183476    6284 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-ca-configmap\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:37.189008 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.188538    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-kube-api-access-x2njk" pod "869b27f2-7068-4501-8d4b-4aae0e159f5e" (UID: "869b27f2-7068-4501-8d4b-4aae0e159f5e"). InnerVolumeSpecName "kube-api-access-x2njk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:51:37.191605 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.191535    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-kubeconfig" pod "869b27f2-7068-4501-8d4b-4aae0e159f5e" (UID: "869b27f2-7068-4501-8d4b-4aae0e159f5e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:51:37.201271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.194399    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-cacerts" pod "869b27f2-7068-4501-8d4b-4aae0e159f5e" (UID: "869b27f2-7068-4501-8d4b-4aae0e159f5e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:51:37.201271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.194850    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-token" pod "869b27f2-7068-4501-8d4b-4aae0e159f5e" (UID: "869b27f2-7068-4501-8d4b-4aae0e159f5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:51:37.285451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.284851    6284 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/869b27f2-7068-4501-8d4b-4aae0e159f5e-local-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:37.285451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.284896    6284 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-csr-dns-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:37.285451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.284912    6284 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-kubeconfig\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:37.285451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.284928    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2njk\" (UniqueName: \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-kube-api-access-x2njk\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:37.285451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.284963    6284 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/869b27f2-7068-4501-8d4b-4aae0e159f5e-cacerts\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:37.285451 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:37.284978    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/869b27f2-7068-4501-8d4b-4aae0e159f5e-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:38.040428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:38.040391    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"
Nov 02 14:51:38.040428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:38.040415    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w" event={"ID":"869b27f2-7068-4501-8d4b-4aae0e159f5e","Type":"ContainerDied","Data":"f29aaba35d4c2b0559ca81db59e685f1b454574a9f7366100bda3567bd608cf0"}
Nov 02 14:51:38.040889 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:38.040480    6284 scope.go:122] "RemoveContainer" containerID="1490046076bcb7cb42f6588d0deff053a9beca33c4fe23e5c7d50582a20dff02"
Nov 02 14:51:38.096324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:38.096284    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"]
Nov 02 14:51:38.105678 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:38.105639    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n885w"]
Nov 02 14:51:38.177886 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:38.177848    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869b27f2-7068-4501-8d4b-4aae0e159f5e" path="/var/lib/kubelet/pods/869b27f2-7068-4501-8d4b-4aae0e159f5e/volumes"
Nov 02 14:51:42.506028 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:42.505982    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g" podUID="9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:51:43.896356 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:51:43.896316    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:51:52.506000 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:52.505887    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g" podUID="9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:51:53.726115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:53.726081    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:51:53.798819 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:53.798780    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-kube-api-access-k88qk\" (UniqueName: \"kubernetes.io/projected/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-kube-api-access-k88qk\") pod \"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4\" (UID: \"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4\") "
Nov 02 14:51:53.799065 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:53.799046    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-plugins-config-volume\") pod \"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4\" (UID: \"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4\") "
Nov 02 14:51:53.799410 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:53.799385    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-plugins-config-volume" pod "9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4" (UID: "9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:51:53.799590 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:53.799568    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:53.801098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:53.801074    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-kube-api-access-k88qk" pod "9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4" (UID: "9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4"). InnerVolumeSpecName "kube-api-access-k88qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:51:53.900694 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:53.900650    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88qk\" (UniqueName: \"kubernetes.io/projected/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4-kube-api-access-k88qk\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:51:54.148873 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:54.148831    6284 generic.go:356] "Generic (PLEG): container finished" podID="9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4" containerID="69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb" exitCode=0
Nov 02 14:51:54.149119 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:54.148883    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"
Nov 02 14:51:54.149119 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:54.148911    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g" event={"ID":"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4","Type":"ContainerDied","Data":"69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb"}
Nov 02 14:51:54.149119 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:54.149009    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g" event={"ID":"9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4","Type":"ContainerDied","Data":"cb129efeb7a596f7a4c150e7e74df1c3b4956d72f2066ce6673dd26fb1d5a997"}
Nov 02 14:51:54.149119 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:54.149039    6284 scope.go:122] "RemoveContainer" containerID="69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb"
Nov 02 14:51:54.160526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:54.160501    6284 scope.go:122] "RemoveContainer" containerID="69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb"
Nov 02 14:51:54.160904 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:51:54.160877    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb\": container with ID starting with 69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb not found: ID does not exist" containerID="69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb"
Nov 02 14:51:54.161028 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:54.160913    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb"} err="failed to get container status \"69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb\": rpc error: code = NotFound desc = could not find container \"69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb\": container with ID starting with 69fba00b755bb44c72a21d4338df21c22c5d9c253ac75fc7c2c5b831bfc0a1eb not found: ID does not exist"
Nov 02 14:51:54.193980 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:54.193948    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"]
Nov 02 14:51:54.203611 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:54.203570    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2d13f4-679589bc79-wmp2g"]
Nov 02 14:51:56.173258 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:51:56.173223    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4" path="/var/lib/kubelet/pods/9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4/volumes"
Nov 02 14:52:05.655899 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:05.655845    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"]
Nov 02 14:52:05.656487 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:05.656164    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf" podUID="5515bb77-f206-4c6e-9f0f-3f800a408723" containerName="maas-api" containerID="cri-o://beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126" gracePeriod=30
Nov 02 14:52:05.991924 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:05.991890    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:52:06.026438 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.026398    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"]
Nov 02 14:52:06.026714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.026649    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr" podUID="da26804e-7a73-49b0-8610-95f68c2e7845" containerName="maas-api" containerID="cri-o://cc00bfa63aa3f3797eb3e03c47c97c0a77b84445923f1e973691628db26826cc" gracePeriod=30
Nov 02 14:52:06.179405 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.179153    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-metrics-certs\") pod \"5515bb77-f206-4c6e-9f0f-3f800a408723\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") "
Nov 02 14:52:06.179629 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.179521    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5515bb77-f206-4c6e-9f0f-3f800a408723-kube-api-access-b5lpz\" (UniqueName: \"kubernetes.io/projected/5515bb77-f206-4c6e-9f0f-3f800a408723-kube-api-access-b5lpz\") pod \"5515bb77-f206-4c6e-9f0f-3f800a408723\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") "
Nov 02 14:52:06.179629 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.179607    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-maas-api-tls\") pod \"5515bb77-f206-4c6e-9f0f-3f800a408723\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") "
Nov 02 14:52:06.179767 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.179689    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5515bb77-f206-4c6e-9f0f-3f800a408723-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5515bb77-f206-4c6e-9f0f-3f800a408723-openshift-service-ca\") pod \"5515bb77-f206-4c6e-9f0f-3f800a408723\" (UID: \"5515bb77-f206-4c6e-9f0f-3f800a408723\") "
Nov 02 14:52:06.182531 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.182313    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5515bb77-f206-4c6e-9f0f-3f800a408723-openshift-service-ca" pod "5515bb77-f206-4c6e-9f0f-3f800a408723" (UID: "5515bb77-f206-4c6e-9f0f-3f800a408723"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:06.190033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.187744    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr" podUID="da26804e-7a73-49b0-8610-95f68c2e7845" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.125:8443/health\": dial tcp 10.134.0.125:8443: connect: connection refused"
Nov 02 14:52:06.194535 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.194469    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-maas-api-tls" pod "5515bb77-f206-4c6e-9f0f-3f800a408723" (UID: "5515bb77-f206-4c6e-9f0f-3f800a408723"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:52:06.198227 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.198110    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5515bb77-f206-4c6e-9f0f-3f800a408723-kube-api-access-b5lpz" pod "5515bb77-f206-4c6e-9f0f-3f800a408723" (UID: "5515bb77-f206-4c6e-9f0f-3f800a408723"). InnerVolumeSpecName "kube-api-access-b5lpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:06.198818 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.198675    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-metrics-certs" pod "5515bb77-f206-4c6e-9f0f-3f800a408723" (UID: "5515bb77-f206-4c6e-9f0f-3f800a408723"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:52:06.233838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.233796    6284 generic.go:356] "Generic (PLEG): container finished" podID="da26804e-7a73-49b0-8610-95f68c2e7845" containerID="cc00bfa63aa3f3797eb3e03c47c97c0a77b84445923f1e973691628db26826cc" exitCode=0
Nov 02 14:52:06.234229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.233910    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr" event={"ID":"da26804e-7a73-49b0-8610-95f68c2e7845","Type":"ContainerDied","Data":"cc00bfa63aa3f3797eb3e03c47c97c0a77b84445923f1e973691628db26826cc"}
Nov 02 14:52:06.250209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.238244    6284 generic.go:356] "Generic (PLEG): container finished" podID="5515bb77-f206-4c6e-9f0f-3f800a408723" containerID="beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126" exitCode=0
Nov 02 14:52:06.250209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.238507    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf" event={"ID":"5515bb77-f206-4c6e-9f0f-3f800a408723","Type":"ContainerDied","Data":"beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126"}
Nov 02 14:52:06.250209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.238563    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf" event={"ID":"5515bb77-f206-4c6e-9f0f-3f800a408723","Type":"ContainerDied","Data":"8c9835a98ea232bf110b8b685968d0d0bf145872f6e1567d24a4fce3d718309b"}
Nov 02 14:52:06.250209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.238595    6284 scope.go:122] "RemoveContainer" containerID="beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126"
Nov 02 14:52:06.250209 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.238785    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"
Nov 02 14:52:06.264971 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.264902    6284 scope.go:122] "RemoveContainer" containerID="beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126"
Nov 02 14:52:06.265372 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:52:06.265341    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126\": container with ID starting with beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126 not found: ID does not exist" containerID="beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126"
Nov 02 14:52:06.265447 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.265389    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126"} err="failed to get container status \"beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126\": rpc error: code = NotFound desc = could not find container \"beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126\": container with ID starting with beb3c3b66a5bcf15fbdc7a4e3da94683d01b97fbc6d4d8a9dde0b2fc713ec126 not found: ID does not exist"
Nov 02 14:52:06.286956 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.282262    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:06.286956 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.282304    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lpz\" (UniqueName: \"kubernetes.io/projected/5515bb77-f206-4c6e-9f0f-3f800a408723-kube-api-access-b5lpz\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:06.286956 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.282319    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5515bb77-f206-4c6e-9f0f-3f800a408723-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:06.286956 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.282335    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5515bb77-f206-4c6e-9f0f-3f800a408723-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:06.303003 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.296042    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"]
Nov 02 14:52:06.329068 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.329027    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d13f4-594794cc44-2c7bf"]
Nov 02 14:52:06.376836 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.376805    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:52:06.487961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.485363    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da26804e-7a73-49b0-8610-95f68c2e7845-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da26804e-7a73-49b0-8610-95f68c2e7845-openshift-service-ca\") pod \"da26804e-7a73-49b0-8610-95f68c2e7845\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") "
Nov 02 14:52:06.487961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.485533    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da26804e-7a73-49b0-8610-95f68c2e7845-kube-api-access-vmnzb\" (UniqueName: \"kubernetes.io/projected/da26804e-7a73-49b0-8610-95f68c2e7845-kube-api-access-vmnzb\") pod \"da26804e-7a73-49b0-8610-95f68c2e7845\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") "
Nov 02 14:52:06.487961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.485575    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-metrics-certs\") pod \"da26804e-7a73-49b0-8610-95f68c2e7845\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") "
Nov 02 14:52:06.487961 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.485681    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-maas-api-tls\") pod \"da26804e-7a73-49b0-8610-95f68c2e7845\" (UID: \"da26804e-7a73-49b0-8610-95f68c2e7845\") "
Nov 02 14:52:06.509598 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.509202    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da26804e-7a73-49b0-8610-95f68c2e7845-openshift-service-ca" pod "da26804e-7a73-49b0-8610-95f68c2e7845" (UID: "da26804e-7a73-49b0-8610-95f68c2e7845"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:06.512783 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.512107    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da26804e-7a73-49b0-8610-95f68c2e7845-kube-api-access-vmnzb" pod "da26804e-7a73-49b0-8610-95f68c2e7845" (UID: "da26804e-7a73-49b0-8610-95f68c2e7845"). InnerVolumeSpecName "kube-api-access-vmnzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:06.514308 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.514195    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-metrics-certs" pod "da26804e-7a73-49b0-8610-95f68c2e7845" (UID: "da26804e-7a73-49b0-8610-95f68c2e7845"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:52:06.514669 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.514642    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-maas-api-tls" pod "da26804e-7a73-49b0-8610-95f68c2e7845" (UID: "da26804e-7a73-49b0-8610-95f68c2e7845"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:52:06.600383 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.600349    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmnzb\" (UniqueName: \"kubernetes.io/projected/da26804e-7a73-49b0-8610-95f68c2e7845-kube-api-access-vmnzb\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:06.600383 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.600379    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:06.600383 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.600390    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da26804e-7a73-49b0-8610-95f68c2e7845-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:06.600650 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:06.600398    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da26804e-7a73-49b0-8610-95f68c2e7845-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:07.246706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:07.246671    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"
Nov 02 14:52:07.246706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:07.246681    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr" event={"ID":"da26804e-7a73-49b0-8610-95f68c2e7845","Type":"ContainerDied","Data":"1af7c16a42d443e6754958188db77b7d15a2e0e6381b0bb8a2d7b69340db346c"}
Nov 02 14:52:07.247293 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:07.246756    6284 scope.go:122] "RemoveContainer" containerID="cc00bfa63aa3f3797eb3e03c47c97c0a77b84445923f1e973691628db26826cc"
Nov 02 14:52:07.293966 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:07.293907    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"]
Nov 02 14:52:07.303560 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:07.303500    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-bc34db-6955845db9-jvswr"]
Nov 02 14:52:08.170324 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:08.170289    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5515bb77-f206-4c6e-9f0f-3f800a408723" path="/var/lib/kubelet/pods/5515bb77-f206-4c6e-9f0f-3f800a408723/volumes"
Nov 02 14:52:08.170813 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:08.170798    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da26804e-7a73-49b0-8610-95f68c2e7845" path="/var/lib/kubelet/pods/da26804e-7a73-49b0-8610-95f68c2e7845/volumes"
Nov 02 14:52:13.895599 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:52:13.895555    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:52:23.181995 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.181948    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"]
Nov 02 14:52:23.182551 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.182226    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8" podUID="897df9cf-6a88-42b3-9f22-61dcf942c079" containerName="maas-api" containerID="cri-o://61a7b961655295e805fc0bbdebd345375de6136a7614adc6b3ee458125b9c868" gracePeriod=30
Nov 02 14:52:23.358630 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.358593    6284 generic.go:356] "Generic (PLEG): container finished" podID="897df9cf-6a88-42b3-9f22-61dcf942c079" containerID="61a7b961655295e805fc0bbdebd345375de6136a7614adc6b3ee458125b9c868" exitCode=0
Nov 02 14:52:23.358856 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.358773    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8" event={"ID":"897df9cf-6a88-42b3-9f22-61dcf942c079","Type":"ContainerDied","Data":"61a7b961655295e805fc0bbdebd345375de6136a7614adc6b3ee458125b9c868"}
Nov 02 14:52:23.489569 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.489538    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:52:23.617494 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.617454    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-metrics-certs\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-metrics-certs\") pod \"897df9cf-6a88-42b3-9f22-61dcf942c079\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") "
Nov 02 14:52:23.617797 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.617652    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/897df9cf-6a88-42b3-9f22-61dcf942c079-kube-api-access-cpmqs\" (UniqueName: \"kubernetes.io/projected/897df9cf-6a88-42b3-9f22-61dcf942c079-kube-api-access-cpmqs\") pod \"897df9cf-6a88-42b3-9f22-61dcf942c079\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") "
Nov 02 14:52:23.617797 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.617689    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/897df9cf-6a88-42b3-9f22-61dcf942c079-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897df9cf-6a88-42b3-9f22-61dcf942c079-openshift-service-ca\") pod \"897df9cf-6a88-42b3-9f22-61dcf942c079\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") "
Nov 02 14:52:23.617797 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.617714    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-maas-api-tls\") pod \"897df9cf-6a88-42b3-9f22-61dcf942c079\" (UID: \"897df9cf-6a88-42b3-9f22-61dcf942c079\") "
Nov 02 14:52:23.618155 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.618130    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897df9cf-6a88-42b3-9f22-61dcf942c079-openshift-service-ca" pod "897df9cf-6a88-42b3-9f22-61dcf942c079" (UID: "897df9cf-6a88-42b3-9f22-61dcf942c079"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:23.619930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.619903    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-metrics-certs" pod "897df9cf-6a88-42b3-9f22-61dcf942c079" (UID: "897df9cf-6a88-42b3-9f22-61dcf942c079"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:52:23.620062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.619928    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-maas-api-tls" pod "897df9cf-6a88-42b3-9f22-61dcf942c079" (UID: "897df9cf-6a88-42b3-9f22-61dcf942c079"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:52:23.620062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.619979    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897df9cf-6a88-42b3-9f22-61dcf942c079-kube-api-access-cpmqs" pod "897df9cf-6a88-42b3-9f22-61dcf942c079" (UID: "897df9cf-6a88-42b3-9f22-61dcf942c079"). InnerVolumeSpecName "kube-api-access-cpmqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:23.720043 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.719917    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpmqs\" (UniqueName: \"kubernetes.io/projected/897df9cf-6a88-42b3-9f22-61dcf942c079-kube-api-access-cpmqs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:23.720043 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.719989    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/897df9cf-6a88-42b3-9f22-61dcf942c079-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:23.720043 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.720001    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:23.720043 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:23.720012    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/897df9cf-6a88-42b3-9f22-61dcf942c079-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:52:24.366297 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:24.366243    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8" event={"ID":"897df9cf-6a88-42b3-9f22-61dcf942c079","Type":"ContainerDied","Data":"e5fbcbc82207c327002d680c0bf6d3cfe7a59a9e38abe183427815249c874395"}
Nov 02 14:52:24.366297 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:24.366282    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"
Nov 02 14:52:24.366875 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:24.366325    6284 scope.go:122] "RemoveContainer" containerID="61a7b961655295e805fc0bbdebd345375de6136a7614adc6b3ee458125b9c868"
Nov 02 14:52:24.411460 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:24.411415    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"]
Nov 02 14:52:24.421064 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:24.421018    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-605541-85686d6868-kb2r8"]
Nov 02 14:52:26.170488 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:26.170453    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="897df9cf-6a88-42b3-9f22-61dcf942c079" path="/var/lib/kubelet/pods/897df9cf-6a88-42b3-9f22-61dcf942c079/volumes"
Nov 02 14:52:43.896678 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:52:43.896634    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:52:45.943505 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.943463    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"]
Nov 02 14:52:45.944431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944403    6284 cpu_manager.go:414] "containerMap: removing container" podUID="869b27f2-7068-4501-8d4b-4aae0e159f5e" containerName="discovery"
Nov 02 14:52:45.944431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944431    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869b27f2-7068-4501-8d4b-4aae0e159f5e" containerName="discovery"
Nov 02 14:52:45.944431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944445    6284 cpu_manager.go:414] "containerMap: removing container" podUID="5515bb77-f206-4c6e-9f0f-3f800a408723" containerName="maas-api"
Nov 02 14:52:45.944704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944453    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5515bb77-f206-4c6e-9f0f-3f800a408723" containerName="maas-api"
Nov 02 14:52:45.944704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944488    6284 cpu_manager.go:414] "containerMap: removing container" podUID="da26804e-7a73-49b0-8610-95f68c2e7845" containerName="maas-api"
Nov 02 14:52:45.944704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944496    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da26804e-7a73-49b0-8610-95f68c2e7845" containerName="maas-api"
Nov 02 14:52:45.944704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944513    6284 cpu_manager.go:414] "containerMap: removing container" podUID="daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50" containerName="maas-api"
Nov 02 14:52:45.944704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944519    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50" containerName="maas-api"
Nov 02 14:52:45.944704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944532    6284 cpu_manager.go:414] "containerMap: removing container" podUID="897df9cf-6a88-42b3-9f22-61dcf942c079" containerName="maas-api"
Nov 02 14:52:45.944704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944538    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="897df9cf-6a88-42b3-9f22-61dcf942c079" containerName="maas-api"
Nov 02 14:52:45.944704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944555    6284 cpu_manager.go:414] "containerMap: removing container" podUID="9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4" containerName="payload-processing"
Nov 02 14:52:45.944704 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944565    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4" containerName="payload-processing"
Nov 02 14:52:45.945201 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944710    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="869b27f2-7068-4501-8d4b-4aae0e159f5e" containerName="discovery"
Nov 02 14:52:45.945201 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944723    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="5515bb77-f206-4c6e-9f0f-3f800a408723" containerName="maas-api"
Nov 02 14:52:45.945201 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944737    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="da26804e-7a73-49b0-8610-95f68c2e7845" containerName="maas-api"
Nov 02 14:52:45.945201 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944749    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="daebe7ed-d010-4e35-a7f5-4a1aaf5e0c50" containerName="maas-api"
Nov 02 14:52:45.945201 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944760    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="897df9cf-6a88-42b3-9f22-61dcf942c079" containerName="maas-api"
Nov 02 14:52:45.945201 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.944770    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a2d075c-f2bb-4aeb-a32a-ef78bfae41c4" containerName="payload-processing"
Nov 02 14:52:45.950490 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.950311    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:45.973016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.972979    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-21baa7\"/\"rate-test-model-21baa7-kserve-self-signed-certs\""
Nov 02 14:52:45.973259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.972994    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-21baa7\"/\"openshift-service-ca.crt\""
Nov 02 14:52:45.973259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.973004    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-21baa7\"/\"default-dockercfg-kb7wh\""
Nov 02 14:52:45.973405 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.973071    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-21baa7\"/\"kube-root-ca.crt\""
Nov 02 14:52:45.976597 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:45.976566    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"]
Nov 02 14:52:46.130822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.130778    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-model-cache\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.130822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.130837    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-home\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.131144 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.130923    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tls-certs\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.131144 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.131097    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk69c\" (UniqueName: \"kubernetes.io/projected/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-kube-api-access-hk69c\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.131279 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.131176    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tmp-dir\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.131279 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.131224    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-dshm\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.232824 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.232706    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk69c\" (UniqueName: \"kubernetes.io/projected/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-kube-api-access-hk69c\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.233052 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.232841    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tmp-dir\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.233052 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.232876    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-dshm\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.233052 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.232974    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-model-cache\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.233052 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.233025    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-home\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.233309 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.233056    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tls-certs\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.233818 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.233788    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-model-cache\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.234028 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.233818    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-home\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.234154 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.234084    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tmp-dir\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.235645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.235616    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-dshm\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.236097 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.236073    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tls-certs\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.255431 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.254745    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk69c\" (UniqueName: \"kubernetes.io/projected/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-kube-api-access-hk69c\") pod \"rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") " pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.266422 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.266383    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:46.583838 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:46.583809    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"]
Nov 02 14:52:46.586999 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:52:46.586964    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaeba8f7_5d2a_47cc_8138_0ae3e4c8fffd.slice/crio-9cb7a19f09fba732d23a3d260227ff922899ecb4d5f5ec46e48e68b908ad5995 WatchSource:0}: Error finding container 9cb7a19f09fba732d23a3d260227ff922899ecb4d5f5ec46e48e68b908ad5995: Status 404 returned error can't find the container with id 9cb7a19f09fba732d23a3d260227ff922899ecb4d5f5ec46e48e68b908ad5995
Nov 02 14:52:47.535156 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:47.535113    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx" event={"ID":"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd","Type":"ContainerStarted","Data":"b6938e2b280e4c9b2b6163ddbabcbec5104682d9d8876cbca4ee006a7dbf1017"}
Nov 02 14:52:47.535667 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:47.535165    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx" event={"ID":"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd","Type":"ContainerStarted","Data":"9cb7a19f09fba732d23a3d260227ff922899ecb4d5f5ec46e48e68b908ad5995"}
Nov 02 14:52:47.572986 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:47.572893    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx" podStartSLOduration=2.572871139 podStartE2EDuration="2.572871139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:52:47.572133376 +0000 UTC m=+4099.531508984" watchObservedRunningTime="2026-11-02 14:52:47.572871139 +0000 UTC m=+4099.532246743"
Nov 02 14:52:48.267037 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:48.267000    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:56.267695 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:56.267653    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:56.281530 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:56.281492    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:52:56.608326 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:52:56.608230    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:53:00.523372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:00.523316    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"]
Nov 02 14:53:00.523926 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:00.523645    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42" podUID="d25add76-ab8b-4fee-8a5d-484fc0ce58ff" containerName="maas-api" containerID="cri-o://071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7" gracePeriod=30
Nov 02 14:53:00.878945 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:00.878896    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:53:00.963371 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:00.963327    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-metrics-certs\") pod \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") "
Nov 02 14:53:00.963592 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:00.963442    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-maas-api-tls\") pod \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") "
Nov 02 14:53:00.963592 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:00.963491    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-kube-api-access-7rs7n\" (UniqueName: \"kubernetes.io/projected/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-kube-api-access-7rs7n\") pod \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") "
Nov 02 14:53:00.963736 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:00.963718    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-openshift-service-ca\") pod \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\" (UID: \"d25add76-ab8b-4fee-8a5d-484fc0ce58ff\") "
Nov 02 14:53:00.990236 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:00.990167    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-maas-api-tls" pod "d25add76-ab8b-4fee-8a5d-484fc0ce58ff" (UID: "d25add76-ab8b-4fee-8a5d-484fc0ce58ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:53:00.996103 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:00.996040    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-kube-api-access-7rs7n" pod "d25add76-ab8b-4fee-8a5d-484fc0ce58ff" (UID: "d25add76-ab8b-4fee-8a5d-484fc0ce58ff"). InnerVolumeSpecName "kube-api-access-7rs7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:00.996759 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:00.996722    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-metrics-certs" pod "d25add76-ab8b-4fee-8a5d-484fc0ce58ff" (UID: "d25add76-ab8b-4fee-8a5d-484fc0ce58ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:53:01.004225 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.004185    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-openshift-service-ca" pod "d25add76-ab8b-4fee-8a5d-484fc0ce58ff" (UID: "d25add76-ab8b-4fee-8a5d-484fc0ce58ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:01.065269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.065224    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.065269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.065262    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rs7n\" (UniqueName: \"kubernetes.io/projected/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-kube-api-access-7rs7n\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.065269 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.065276    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.065544 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.065291    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d25add76-ab8b-4fee-8a5d-484fc0ce58ff-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.686570 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.686525    6284 generic.go:356] "Generic (PLEG): container finished" podID="d25add76-ab8b-4fee-8a5d-484fc0ce58ff" containerID="071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7" exitCode=0
Nov 02 14:53:01.687089 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.686589    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"
Nov 02 14:53:01.687089 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.686608    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42" event={"ID":"d25add76-ab8b-4fee-8a5d-484fc0ce58ff","Type":"ContainerDied","Data":"071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7"}
Nov 02 14:53:01.687089 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.686671    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42" event={"ID":"d25add76-ab8b-4fee-8a5d-484fc0ce58ff","Type":"ContainerDied","Data":"f81a33bc5dd04601521a4f7bd2ec0246e87d4ebf9043d55704be2bd1e5eb0d91"}
Nov 02 14:53:01.687089 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.686698    6284 scope.go:122] "RemoveContainer" containerID="071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7"
Nov 02 14:53:01.699661 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.699638    6284 scope.go:122] "RemoveContainer" containerID="071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7"
Nov 02 14:53:01.700060 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:53:01.700031    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7\": container with ID starting with 071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7 not found: ID does not exist" containerID="071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7"
Nov 02 14:53:01.700155 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.700074    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7"} err="failed to get container status \"071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7\": rpc error: code = NotFound desc = could not find container \"071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7\": container with ID starting with 071f1735d449f997b79451131b2ec9f1967201b9450420e131b664d200dab5b7 not found: ID does not exist"
Nov 02 14:53:01.733545 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.733507    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"]
Nov 02 14:53:01.743180 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:01.743135    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4b7759-865c589778-7vc42"]
Nov 02 14:53:02.170208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:02.170169    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25add76-ab8b-4fee-8a5d-484fc0ce58ff" path="/var/lib/kubelet/pods/d25add76-ab8b-4fee-8a5d-484fc0ce58ff/volumes"
Nov 02 14:53:09.431418 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.431372    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"]
Nov 02 14:53:09.432283 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.432258    6284 cpu_manager.go:414] "containerMap: removing container" podUID="d25add76-ab8b-4fee-8a5d-484fc0ce58ff" containerName="maas-api"
Nov 02 14:53:09.432283 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.432282    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25add76-ab8b-4fee-8a5d-484fc0ce58ff" containerName="maas-api"
Nov 02 14:53:09.432557 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.432464    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25add76-ab8b-4fee-8a5d-484fc0ce58ff" containerName="maas-api"
Nov 02 14:53:09.437800 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.437764    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.460080 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.460046    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-2cddf2\"/\"openshift-service-ca.crt\""
Nov 02 14:53:09.460080 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.460059    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-2cddf2\"/\"kube-root-ca.crt\""
Nov 02 14:53:09.460433 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.460281    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-2cddf2\"/\"rate-test-model-2cddf2-kserve-self-signed-certs\""
Nov 02 14:53:09.460433 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.460330    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-2cddf2\"/\"default-dockercfg-v4fzf\""
Nov 02 14:53:09.475242 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.475187    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"]
Nov 02 14:53:09.496824 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.496774    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tls-certs\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.497074 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.496862    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qms6x\" (UniqueName: \"kubernetes.io/projected/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-kube-api-access-qms6x\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.497074 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.497014    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-home\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.497194 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.497074    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-model-cache\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.497235 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.497194    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-dshm\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.497359 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.497337    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tmp-dir\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.598802 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.598752    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-model-cache\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.599062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.598846    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-dshm\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.599062 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.598981    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tmp-dir\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.599193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.599105    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tls-certs\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.599193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.599159    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qms6x\" (UniqueName: \"kubernetes.io/projected/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-kube-api-access-qms6x\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.599302 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.599205    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-home\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.599302 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.599280    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-model-cache\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.599573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.599549    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-home\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.599749 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.599683    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tmp-dir\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.601692 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.601631    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-dshm\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.601968 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.601911    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tls-certs\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.623304 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.623267    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qms6x\" (UniqueName: \"kubernetes.io/projected/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-kube-api-access-qms6x\") pod \"rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") " pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:09.753106 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:09.753064    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:10.083286 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:10.083240    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"]
Nov 02 14:53:10.085325 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:53:10.085285    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1af7bc01_2fb9_45c4_9b6e_1e9ee016bda0.slice/crio-a39f686871cf72393be55cda71bcfc39cca80b249a0fbd93571206e300e58199 WatchSource:0}: Error finding container a39f686871cf72393be55cda71bcfc39cca80b249a0fbd93571206e300e58199: Status 404 returned error can't find the container with id a39f686871cf72393be55cda71bcfc39cca80b249a0fbd93571206e300e58199
Nov 02 14:53:10.756367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:10.756330    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb" event={"ID":"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0","Type":"ContainerStarted","Data":"36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d"}
Nov 02 14:53:10.756367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:10.756380    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb" event={"ID":"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0","Type":"ContainerStarted","Data":"a39f686871cf72393be55cda71bcfc39cca80b249a0fbd93571206e300e58199"}
Nov 02 14:53:10.800893 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:10.800832    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb" podStartSLOduration=1.800815461 podStartE2EDuration="1.800815461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:10.797607726 +0000 UTC m=+4122.756983332" watchObservedRunningTime="2026-11-02 14:53:10.800815461 +0000 UTC m=+4122.760191069"
Nov 02 14:53:11.753270 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:11.753223    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:13.927196 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:53:13.927152    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:53:19.753292 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:19.753248    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:19.769011 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:19.768973    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:19.841039 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:19.841006    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:53:23.965564 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:23.965473    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"]
Nov 02 14:53:23.969991 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:23.969952    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:23.987261 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:23.987220    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-0d1ca7\"/\"kube-root-ca.crt\""
Nov 02 14:53:23.987421 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:23.987316    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-0d1ca7\"/\"openshift-service-ca.crt\""
Nov 02 14:53:23.997347 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:23.997310    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-0d1ca7\"/\"default-dockercfg-lbn2c\""
Nov 02 14:53:23.997573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:23.997347    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-0d1ca7\"/\"rate-test-model-0d1ca7-kserve-self-signed-certs\""
Nov 02 14:53:24.002234 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.002198    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"]
Nov 02 14:53:24.104066 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.104011    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-dshm\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.104285 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.104184    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tmp-dir\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.104412 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.104381    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-home\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.104504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.104475    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tls-certs\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.104619 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.104530    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-model-cache\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.104695 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.104670    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7ndx\" (UniqueName: \"kubernetes.io/projected/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-kube-api-access-g7ndx\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.206253 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.206213    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-home\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.206424 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.206285    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tls-certs\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.206424 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.206330    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-model-cache\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.206540 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.206425    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7ndx\" (UniqueName: \"kubernetes.io/projected/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-kube-api-access-g7ndx\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.206540 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.206493    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-dshm\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.206622 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.206551    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tmp-dir\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.206711 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.206686    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-home\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.207000 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.206980    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-model-cache\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.207047 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.206979    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tmp-dir\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.210336 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.210303    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tls-certs\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.231587 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.231492    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7ndx\" (UniqueName: \"kubernetes.io/projected/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-kube-api-access-g7ndx\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.236984 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.236385    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-dshm\") pod \"rate-test-model-0d1ca7-kserve-b684dc889-jwktr\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") " pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.286456 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.286407    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:24.603665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.602642    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"]
Nov 02 14:53:24.606521 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:53:24.606480    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadbf61d4_f51d_4ef7_8eda_c2a88a0ad96e.slice/crio-029dd9d7d171b209d09a1c6b274aa218531562827b23ed15654f558c28bc08f6 WatchSource:0}: Error finding container 029dd9d7d171b209d09a1c6b274aa218531562827b23ed15654f558c28bc08f6: Status 404 returned error can't find the container with id 029dd9d7d171b209d09a1c6b274aa218531562827b23ed15654f558c28bc08f6
Nov 02 14:53:24.864556 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.864460    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr" event={"ID":"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e","Type":"ContainerStarted","Data":"42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb"}
Nov 02 14:53:24.864556 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.864514    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr" event={"ID":"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e","Type":"ContainerStarted","Data":"029dd9d7d171b209d09a1c6b274aa218531562827b23ed15654f558c28bc08f6"}
Nov 02 14:53:24.910211 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:24.910147    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr" podStartSLOduration=1.910130869 podStartE2EDuration="1.910130869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:24.909093668 +0000 UTC m=+4136.868469276" watchObservedRunningTime="2026-11-02 14:53:24.910130869 +0000 UTC m=+4136.869506475"
Nov 02 14:53:25.286532 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:25.286493    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:34.287533 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:34.287496    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:34.301077 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:34.301037    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:34.943232 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:34.943200    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:53:39.368826 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.368784    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"]
Nov 02 14:53:39.374774 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.374745    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.394051 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.394018    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"]
Nov 02 14:53:39.398782 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.398744    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4c2811\"/\"openshift-service-ca.crt\""
Nov 02 14:53:39.399057 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.398752    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4c2811\"/\"kube-root-ca.crt\""
Nov 02 14:53:39.437908 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.437861    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4c2811\"/\"default-dockercfg-ww7fw\""
Nov 02 14:53:39.438133 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.438035    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4c2811\"/\"rate-test-model-4c2811-kserve-self-signed-certs\""
Nov 02 14:53:39.507005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.506958    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-tmp-dir\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.507233 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.507026    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfrgm\" (UniqueName: \"kubernetes.io/projected/261ca16d-584d-4b15-b34c-0a54eb966013-kube-api-access-sfrgm\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.507233 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.507057    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-dshm\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.507233 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.507098    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-home\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.507233 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.507205    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-model-cache\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.507500 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.507403    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261ca16d-584d-4b15-b34c-0a54eb966013-tls-certs\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.616830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.615721    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261ca16d-584d-4b15-b34c-0a54eb966013-tls-certs\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.616830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.615864    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-tmp-dir\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.616830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.615901    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfrgm\" (UniqueName: \"kubernetes.io/projected/261ca16d-584d-4b15-b34c-0a54eb966013-kube-api-access-sfrgm\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.616830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.615948    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-dshm\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.616830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.615984    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-home\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.616830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.616030    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-model-cache\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.616830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.616761    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-model-cache\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.617434 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.617038    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-tmp-dir\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.617434 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.617090    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-home\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.622043 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.621967    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261ca16d-584d-4b15-b34c-0a54eb966013-tls-certs\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.627534 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.627460    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-dshm\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.642843 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.642766    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfrgm\" (UniqueName: \"kubernetes.io/projected/261ca16d-584d-4b15-b34c-0a54eb966013-kube-api-access-sfrgm\") pod \"rate-test-model-4c2811-kserve-78986bc477-74p58\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") " pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:39.691302 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:39.690920    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:40.038235 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:40.038201    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"]
Nov 02 14:53:40.041362 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:53:40.041327    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod261ca16d_584d_4b15_b34c_0a54eb966013.slice/crio-32a9ed6bbd1752cfaf7de961ef7c87b6ce24ea8b7da8ead2fb7bf059676475b8 WatchSource:0}: Error finding container 32a9ed6bbd1752cfaf7de961ef7c87b6ce24ea8b7da8ead2fb7bf059676475b8: Status 404 returned error can't find the container with id 32a9ed6bbd1752cfaf7de961ef7c87b6ce24ea8b7da8ead2fb7bf059676475b8
Nov 02 14:53:40.976271 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:40.976232    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58" event={"ID":"261ca16d-584d-4b15-b34c-0a54eb966013","Type":"ContainerStarted","Data":"762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025"}
Nov 02 14:53:40.976709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:40.976283    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58" event={"ID":"261ca16d-584d-4b15-b34c-0a54eb966013","Type":"ContainerStarted","Data":"32a9ed6bbd1752cfaf7de961ef7c87b6ce24ea8b7da8ead2fb7bf059676475b8"}
Nov 02 14:53:41.020760 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:41.020699    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58" podStartSLOduration=2.020680215 podStartE2EDuration="2.020680215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:41.019323892 +0000 UTC m=+4152.978699499" watchObservedRunningTime="2026-11-02 14:53:41.020680215 +0000 UTC m=+4152.980055858"
Nov 02 14:53:41.692005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:41.691965    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:43.897175 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:53:43.897129    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:53:45.918799 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:45.918758    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"]
Nov 02 14:53:45.928238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:45.928200    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"]
Nov 02 14:53:49.691292 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:49.691246    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:49.704756 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:49.704724    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:53:50.063993 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:53:50.063958    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:54:00.940992 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:00.940889    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main" containerID="cri-o://2216cf8a8c3a18028461f3053086f5d512566825870a0dadca04fdff8d401069" gracePeriod=45
Nov 02 14:54:00.952903 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:00.952852    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main" containerID="cri-o://3ca49d79caefd8eacf94da8dc2c850cac685588ba1276ac1e3e029e21f5a5665" gracePeriod=45
Nov 02 14:54:01.150218 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.150175    6284 generic.go:356] "Generic (PLEG): container finished" podID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerID="3ca49d79caefd8eacf94da8dc2c850cac685588ba1276ac1e3e029e21f5a5665" exitCode=0
Nov 02 14:54:01.150432 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.150222    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" event={"ID":"fca46c70-2ef9-4179-be04-6469e731b1d8","Type":"ContainerDied","Data":"3ca49d79caefd8eacf94da8dc2c850cac685588ba1276ac1e3e029e21f5a5665"}
Nov 02 14:54:01.152308 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.152276    6284 generic.go:356] "Generic (PLEG): container finished" podID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerID="2216cf8a8c3a18028461f3053086f5d512566825870a0dadca04fdff8d401069" exitCode=0
Nov 02 14:54:01.152455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.152345    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" event={"ID":"062cbb9f-9666-434b-bff9-d1e976cdd550","Type":"ContainerDied","Data":"2216cf8a8c3a18028461f3053086f5d512566825870a0dadca04fdff8d401069"}
Nov 02 14:54:01.242822 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.242796    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:54:01.263022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.262987    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:54:01.310644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.310611    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-dshm\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-dshm\") pod \"062cbb9f-9666-434b-bff9-d1e976cdd550\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") "
Nov 02 14:54:01.310870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.310756    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-model-cache\") pod \"062cbb9f-9666-434b-bff9-d1e976cdd550\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") "
Nov 02 14:54:01.310870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.310778    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-tmp-dir\") pod \"062cbb9f-9666-434b-bff9-d1e976cdd550\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") "
Nov 02 14:54:01.310870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.310845    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/062cbb9f-9666-434b-bff9-d1e976cdd550-kube-api-access-gwphz\" (UniqueName: \"kubernetes.io/projected/062cbb9f-9666-434b-bff9-d1e976cdd550-kube-api-access-gwphz\") pod \"062cbb9f-9666-434b-bff9-d1e976cdd550\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") "
Nov 02 14:54:01.311115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.310884    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062cbb9f-9666-434b-bff9-d1e976cdd550-tls-certs\" (UniqueName: \"kubernetes.io/secret/062cbb9f-9666-434b-bff9-d1e976cdd550-tls-certs\") pod \"062cbb9f-9666-434b-bff9-d1e976cdd550\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") "
Nov 02 14:54:01.311115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.310949    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-home\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-home\") pod \"062cbb9f-9666-434b-bff9-d1e976cdd550\" (UID: \"062cbb9f-9666-434b-bff9-d1e976cdd550\") "
Nov 02 14:54:01.311239 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.311137    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-model-cache" pod "062cbb9f-9666-434b-bff9-d1e976cdd550" (UID: "062cbb9f-9666-434b-bff9-d1e976cdd550"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.311417 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.311357    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-tmp-dir" pod "062cbb9f-9666-434b-bff9-d1e976cdd550" (UID: "062cbb9f-9666-434b-bff9-d1e976cdd550"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.311801 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.311531    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-home" pod "062cbb9f-9666-434b-bff9-d1e976cdd550" (UID: "062cbb9f-9666-434b-bff9-d1e976cdd550"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.312241 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.312189    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.312478 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.312438    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.312478 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.312467    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.313595 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.313571    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-dshm" pod "062cbb9f-9666-434b-bff9-d1e976cdd550" (UID: "062cbb9f-9666-434b-bff9-d1e976cdd550"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.313695 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.313593    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062cbb9f-9666-434b-bff9-d1e976cdd550-kube-api-access-gwphz" pod "062cbb9f-9666-434b-bff9-d1e976cdd550" (UID: "062cbb9f-9666-434b-bff9-d1e976cdd550"). InnerVolumeSpecName "kube-api-access-gwphz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:01.313826 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.313795    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062cbb9f-9666-434b-bff9-d1e976cdd550-tls-certs" pod "062cbb9f-9666-434b-bff9-d1e976cdd550" (UID: "062cbb9f-9666-434b-bff9-d1e976cdd550"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:54:01.413369 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.413335    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-dshm\") pod \"fca46c70-2ef9-4179-be04-6469e731b1d8\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") "
Nov 02 14:54:01.413648 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.413387    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca46c70-2ef9-4179-be04-6469e731b1d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/fca46c70-2ef9-4179-be04-6469e731b1d8-tls-certs\") pod \"fca46c70-2ef9-4179-be04-6469e731b1d8\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") "
Nov 02 14:54:01.413648 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.413465    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-model-cache\") pod \"fca46c70-2ef9-4179-be04-6469e731b1d8\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") "
Nov 02 14:54:01.413648 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.413540    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca46c70-2ef9-4179-be04-6469e731b1d8-kube-api-access-tqwpt\" (UniqueName: \"kubernetes.io/projected/fca46c70-2ef9-4179-be04-6469e731b1d8-kube-api-access-tqwpt\") pod \"fca46c70-2ef9-4179-be04-6469e731b1d8\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") "
Nov 02 14:54:01.413648 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.413576    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-home\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-home\") pod \"fca46c70-2ef9-4179-be04-6469e731b1d8\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") "
Nov 02 14:54:01.413910 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.413697    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-tmp-dir\") pod \"fca46c70-2ef9-4179-be04-6469e731b1d8\" (UID: \"fca46c70-2ef9-4179-be04-6469e731b1d8\") "
Nov 02 14:54:01.413910 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.413822    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-home" pod "fca46c70-2ef9-4179-be04-6469e731b1d8" (UID: "fca46c70-2ef9-4179-be04-6469e731b1d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.413910 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.413840    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-model-cache" pod "fca46c70-2ef9-4179-be04-6469e731b1d8" (UID: "fca46c70-2ef9-4179-be04-6469e731b1d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.414121 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.413924    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-tmp-dir" pod "fca46c70-2ef9-4179-be04-6469e731b1d8" (UID: "fca46c70-2ef9-4179-be04-6469e731b1d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.414289 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.414269    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/062cbb9f-9666-434b-bff9-d1e976cdd550-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.414360 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.414293    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.414360 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.414307    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.414360 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.414321    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.414360 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.414335    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwphz\" (UniqueName: \"kubernetes.io/projected/062cbb9f-9666-434b-bff9-d1e976cdd550-kube-api-access-gwphz\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.414360 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.414348    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062cbb9f-9666-434b-bff9-d1e976cdd550-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.415581 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.415550    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-dshm" pod "fca46c70-2ef9-4179-be04-6469e731b1d8" (UID: "fca46c70-2ef9-4179-be04-6469e731b1d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.415664 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.415613    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca46c70-2ef9-4179-be04-6469e731b1d8-tls-certs" pod "fca46c70-2ef9-4179-be04-6469e731b1d8" (UID: "fca46c70-2ef9-4179-be04-6469e731b1d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:54:01.415709 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.415670    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca46c70-2ef9-4179-be04-6469e731b1d8-kube-api-access-tqwpt" pod "fca46c70-2ef9-4179-be04-6469e731b1d8" (UID: "fca46c70-2ef9-4179-be04-6469e731b1d8"). InnerVolumeSpecName "kube-api-access-tqwpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:01.515510 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.515465    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqwpt\" (UniqueName: \"kubernetes.io/projected/fca46c70-2ef9-4179-be04-6469e731b1d8-kube-api-access-tqwpt\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.515510 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.515502    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca46c70-2ef9-4179-be04-6469e731b1d8-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.515510 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:01.515512    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca46c70-2ef9-4179-be04-6469e731b1d8-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.161665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:02.161636    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"
Nov 02 14:54:02.161665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:02.161642    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95" event={"ID":"fca46c70-2ef9-4179-be04-6469e731b1d8","Type":"ContainerDied","Data":"05c465d895c14a5133b74f249efcd9e987e5538b1c6dcfd5a5bf07e567dc738c"}
Nov 02 14:54:02.162265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:02.161724    6284 scope.go:122] "RemoveContainer" containerID="3ca49d79caefd8eacf94da8dc2c850cac685588ba1276ac1e3e029e21f5a5665"
Nov 02 14:54:02.163420 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:02.163398    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"
Nov 02 14:54:02.170070 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:02.170039    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p" event={"ID":"062cbb9f-9666-434b-bff9-d1e976cdd550","Type":"ContainerDied","Data":"609c4651960aef5a61abbe94cca144ab03621c3235048566d2136cf0358e2383"}
Nov 02 14:54:02.173418 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:02.173399    6284 scope.go:122] "RemoveContainer" containerID="2216cf8a8c3a18028461f3053086f5d512566825870a0dadca04fdff8d401069"
Nov 02 14:54:02.230604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:02.230511    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"]
Nov 02 14:54:02.240681 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:02.240648    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-w0-1a562c-kserve-54cc7ccd67-9rn95"]
Nov 02 14:54:02.267375 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:02.267335    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"]
Nov 02 14:54:02.277561 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:02.277525    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/distinct-2-w0-1a562c-kserve-58d9f995bd-8cm9p"]
Nov 02 14:54:04.170570 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:04.170529    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" path="/var/lib/kubelet/pods/062cbb9f-9666-434b-bff9-d1e976cdd550/volumes"
Nov 02 14:54:04.171110 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:04.171093    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" path="/var/lib/kubelet/pods/fca46c70-2ef9-4179-be04-6469e731b1d8/volumes"
Nov 02 14:54:12.471582 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.471541    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"]
Nov 02 14:54:12.472133 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.472113    6284 cpu_manager.go:414] "containerMap: removing container" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main"
Nov 02 14:54:12.472133 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.472126    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main"
Nov 02 14:54:12.472252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.472177    6284 cpu_manager.go:414] "containerMap: removing container" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main"
Nov 02 14:54:12.472252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.472182    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main"
Nov 02 14:54:12.472353 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.472260    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="062cbb9f-9666-434b-bff9-d1e976cdd550" containerName="main"
Nov 02 14:54:12.472353 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.472271    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca46c70-2ef9-4179-be04-6469e731b1d8" containerName="main"
Nov 02 14:54:12.477462 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.477435    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.496602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.496567    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fcefcc-openshift-default-dockercfg-r7kkx\""
Nov 02 14:54:12.510658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.510621    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"]
Nov 02 14:54:12.560418 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.560370    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.560638 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.560438    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.560638 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.560537    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.560638 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.560597    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.560806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.560723    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.560806 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.560776    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.560910 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.560856    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7p28\" (UniqueName: \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-kube-api-access-q7p28\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.560910 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.560888    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca340223-2a40-4e94-a02f-8031a515cc1d-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.561043 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.561024    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.661925 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.661868    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.661925 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.661925    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662088    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662131    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662259 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662164    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662400 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662267    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662400 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662351    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662417    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7p28\" (UniqueName: \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-kube-api-access-q7p28\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662453    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca340223-2a40-4e94-a02f-8031a515cc1d-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662603 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662513    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662696 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662667    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662808    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.662917 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.662848    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.663326 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.663298    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca340223-2a40-4e94-a02f-8031a515cc1d-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.665047 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.665023    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.665265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.665241    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.683612 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.683573    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.683796 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.683743    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7p28\" (UniqueName: \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-kube-api-access-q7p28\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:12.792475 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:12.792437    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:13.118422 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:13.118355    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"]
Nov 02 14:54:13.121234 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:54:13.121195    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca340223_2a40_4e94_a02f_8031a515cc1d.slice/crio-c4c61ab87a90984ed70c73abcd43ce4dad1fc621b55ca12c85b261ca04b67376 WatchSource:0}: Error finding container c4c61ab87a90984ed70c73abcd43ce4dad1fc621b55ca12c85b261ca04b67376: Status 404 returned error can't find the container with id c4c61ab87a90984ed70c73abcd43ce4dad1fc621b55ca12c85b261ca04b67376
Nov 02 14:54:13.124031 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:13.123994    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:54:13.124135 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:13.124073    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:54:13.124135 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:13.124101    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:54:13.238959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:13.238878    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" event={"ID":"ca340223-2a40-4e94-a02f-8031a515cc1d","Type":"ContainerStarted","Data":"8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9"}
Nov 02 14:54:13.239140 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:13.239002    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" event={"ID":"ca340223-2a40-4e94-a02f-8031a515cc1d","Type":"ContainerStarted","Data":"c4c61ab87a90984ed70c73abcd43ce4dad1fc621b55ca12c85b261ca04b67376"}
Nov 02 14:54:13.280727 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:13.280660    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" podStartSLOduration=1.280640499 podStartE2EDuration="1.280640499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:13.279735035 +0000 UTC m=+4185.239110642" watchObservedRunningTime="2026-11-02 14:54:13.280640499 +0000 UTC m=+4185.240016107"
Nov 02 14:54:13.793590 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:13.793553    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:13.901456 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:54:13.901412    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:54:14.795355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:14.795313    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 14:54:14.795799 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:14.795389    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 14:54:15.792945 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:15.792872    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 14:54:15.793161 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:15.792956    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 14:54:16.793021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:16.792983    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 14:54:16.793440 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:16.793045    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 14:54:17.793473 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:17.793427    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 14:54:17.793891 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:17.793497    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 14:54:18.793191 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:18.793143    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Nov 02 14:54:18.793402 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:18.793215    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Nov 02 14:54:19.000929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.000888    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"]
Nov 02 14:54:19.001485 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.001187    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447" podUID="673fcfe6-3203-47b7-9578-5f6d4dd96105" containerName="maas-api" containerID="cri-o://2ea089f5e1f88611291d02a9edcdc99e1ec8894ca08172bd62e0d7cb3edf8723" gracePeriod=30
Nov 02 14:54:19.320009 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.319968    6284 generic.go:356] "Generic (PLEG): container finished" podID="673fcfe6-3203-47b7-9578-5f6d4dd96105" containerID="2ea089f5e1f88611291d02a9edcdc99e1ec8894ca08172bd62e0d7cb3edf8723" exitCode=0
Nov 02 14:54:19.320498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.320468    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447" event={"ID":"673fcfe6-3203-47b7-9578-5f6d4dd96105","Type":"ContainerDied","Data":"2ea089f5e1f88611291d02a9edcdc99e1ec8894ca08172bd62e0d7cb3edf8723"}
Nov 02 14:54:19.387664 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.387618    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 14:54:19.461489 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.461437    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673fcfe6-3203-47b7-9578-5f6d4dd96105-kube-api-access-7dlf4\" (UniqueName: \"kubernetes.io/projected/673fcfe6-3203-47b7-9578-5f6d4dd96105-kube-api-access-7dlf4\") pod \"673fcfe6-3203-47b7-9578-5f6d4dd96105\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") "
Nov 02 14:54:19.461699 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.461547    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-metrics-certs\" (UniqueName: \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-metrics-certs\") pod \"673fcfe6-3203-47b7-9578-5f6d4dd96105\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") "
Nov 02 14:54:19.461699 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.461629    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-maas-api-tls\") pod \"673fcfe6-3203-47b7-9578-5f6d4dd96105\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") "
Nov 02 14:54:19.461699 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.461663    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673fcfe6-3203-47b7-9578-5f6d4dd96105-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673fcfe6-3203-47b7-9578-5f6d4dd96105-openshift-service-ca\") pod \"673fcfe6-3203-47b7-9578-5f6d4dd96105\" (UID: \"673fcfe6-3203-47b7-9578-5f6d4dd96105\") "
Nov 02 14:54:19.470508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.463167    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673fcfe6-3203-47b7-9578-5f6d4dd96105-openshift-service-ca" pod "673fcfe6-3203-47b7-9578-5f6d4dd96105" (UID: "673fcfe6-3203-47b7-9578-5f6d4dd96105"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:54:19.480114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.480076    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-maas-api-tls" pod "673fcfe6-3203-47b7-9578-5f6d4dd96105" (UID: "673fcfe6-3203-47b7-9578-5f6d4dd96105"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:54:19.498238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.496422    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-metrics-certs" pod "673fcfe6-3203-47b7-9578-5f6d4dd96105" (UID: "673fcfe6-3203-47b7-9578-5f6d4dd96105"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:54:19.522406 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.522144    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673fcfe6-3203-47b7-9578-5f6d4dd96105-kube-api-access-7dlf4" pod "673fcfe6-3203-47b7-9578-5f6d4dd96105" (UID: "673fcfe6-3203-47b7-9578-5f6d4dd96105"). InnerVolumeSpecName "kube-api-access-7dlf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:19.563355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.563309    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dlf4\" (UniqueName: \"kubernetes.io/projected/673fcfe6-3203-47b7-9578-5f6d4dd96105-kube-api-access-7dlf4\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:19.563355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.563347    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:19.563355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.563356    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673fcfe6-3203-47b7-9578-5f6d4dd96105-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:19.563355 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.563365    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673fcfe6-3203-47b7-9578-5f6d4dd96105-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:19.823416 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.823383    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:19.824134 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.824098    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:19.827729 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:19.827630    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:54:20.327734 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:20.327700    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"
Nov 02 14:54:20.327734 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:20.327716    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447" event={"ID":"673fcfe6-3203-47b7-9578-5f6d4dd96105","Type":"ContainerDied","Data":"f39b1a7181e2abad3cba539ba98d11924ea26eaa651f2717edd6f9f65080bd26"}
Nov 02 14:54:20.328290 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:20.327790    6284 scope.go:122] "RemoveContainer" containerID="2ea089f5e1f88611291d02a9edcdc99e1ec8894ca08172bd62e0d7cb3edf8723"
Nov 02 14:54:20.373260 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:20.373218    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"]
Nov 02 14:54:20.383034 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:20.382996    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1a562c-54f99fdb67-sm447"]
Nov 02 14:54:22.170452 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:22.170414    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673fcfe6-3203-47b7-9578-5f6d4dd96105" path="/var/lib/kubelet/pods/673fcfe6-3203-47b7-9578-5f6d4dd96105/volumes"
Nov 02 14:54:29.267098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:29.267068    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:54:29.278524 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:29.278493    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:54:36.969240 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:36.969198    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"]
Nov 02 14:54:36.970098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:36.970074    6284 cpu_manager.go:414] "containerMap: removing container" podUID="673fcfe6-3203-47b7-9578-5f6d4dd96105" containerName="maas-api"
Nov 02 14:54:36.970098 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:36.970098    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673fcfe6-3203-47b7-9578-5f6d4dd96105" containerName="maas-api"
Nov 02 14:54:36.970286 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:36.970227    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="673fcfe6-3203-47b7-9578-5f6d4dd96105" containerName="maas-api"
Nov 02 14:54:36.975463 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:36.975435    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.002126 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.002086    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"]
Nov 02 14:54:37.002343 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.002306    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.002461 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.002364    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.002526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.002459    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.002627 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.002609    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.002691 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.002664    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.002755 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.002700    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.002810 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.002773    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.002895 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.002881    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hbf\" (UniqueName: \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-kube-api-access-f4hbf\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.002986 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.002922    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.075080 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.075049    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f30173-openshift-default-dockercfg-tw6wb\""
Nov 02 14:54:37.104113 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104066    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hbf\" (UniqueName: \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-kube-api-access-f4hbf\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.104335 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104130    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.104335 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104204    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.104335 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104231    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.104335 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104256    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.104567 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104375    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.104567 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104426    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.104567 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104458    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.104567 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104529    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.104817 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104789    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.105012 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.104979    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.105162 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.105053    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.105162 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.105140    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.105302 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.105232    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.107019 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.106987    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.107242 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.107219    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.125392 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.125352    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.125575 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.125410    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hbf\" (UniqueName: \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-kube-api-access-f4hbf\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.292417 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.292378    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:37.625129 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.624209    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"]
Nov 02 14:54:37.634847 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.634797    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:54:37.635087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.634901    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:54:37.635087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:37.634987    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:54:38.507489 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:38.507445    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt" event={"ID":"c29c8cf7-2a01-4edc-98be-c7bd46bff630","Type":"ContainerStarted","Data":"88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf"}
Nov 02 14:54:38.508081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:38.508052    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt" event={"ID":"c29c8cf7-2a01-4edc-98be-c7bd46bff630","Type":"ContainerStarted","Data":"e0429111c367a75aadc41499df73d94fc80e94daf9ed1e1b334768fcd5583063"}
Nov 02 14:54:38.629673 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:38.601072    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt" podStartSLOduration=2.601047902 podStartE2EDuration="2.601047902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:38.559515446 +0000 UTC m=+4210.518891055" watchObservedRunningTime="2026-11-02 14:54:38.601047902 +0000 UTC m=+4210.560423511"
Nov 02 14:54:39.293541 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:39.293500    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:39.295780 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:39.295748    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 02 14:54:39.296013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:39.295805    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt" podUID="c29c8cf7-2a01-4edc-98be-c7bd46bff630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 02 14:54:39.442508 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:39.442465    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"]
Nov 02 14:54:39.443045 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:39.443008    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy" containerID="cri-o://19e67c7b68465aea7d9b7162a8d3672e86788e55534055eb64802662c5ce4a58" gracePeriod=30
Nov 02 14:54:40.294476 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:40.294433    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 02 14:54:40.295129 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:40.294504    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt" podUID="c29c8cf7-2a01-4edc-98be-c7bd46bff630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 02 14:54:41.293677 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:41.293639    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 02 14:54:41.293882 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:41.293704    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt" podUID="c29c8cf7-2a01-4edc-98be-c7bd46bff630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 02 14:54:42.293056 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:42.293016    6284 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Nov 02 14:54:42.293515 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:42.293094    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt" podUID="c29c8cf7-2a01-4edc-98be-c7bd46bff630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Nov 02 14:54:42.718625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:42.718521    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"]
Nov 02 14:54:42.719088 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:42.718762    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt" podUID="c29c8cf7-2a01-4edc-98be-c7bd46bff630" containerName="istio-proxy" containerID="cri-o://88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf" gracePeriod=30
Nov 02 14:54:43.900458 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:54:43.900404    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:54:44.562615 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.562260    6284 generic.go:356] "Generic (PLEG): container finished" podID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerID="19e67c7b68465aea7d9b7162a8d3672e86788e55534055eb64802662c5ce4a58" exitCode=0
Nov 02 14:54:44.562615 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.562436    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" event={"ID":"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa","Type":"ContainerDied","Data":"19e67c7b68465aea7d9b7162a8d3672e86788e55534055eb64802662c5ce4a58"}
Nov 02 14:54:44.651380 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.651349    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 14:54:44.743841 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.743782    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-token\" (UniqueName: \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-token\") pod \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") "
Nov 02 14:54:44.744117 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.743890    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-podinfo\") pod \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") "
Nov 02 14:54:44.744208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.744114    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-envoy\") pod \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") "
Nov 02 14:54:44.744208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.744159    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-data\") pod \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") "
Nov 02 14:54:44.744327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.744214    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istiod-ca-cert\") pod \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") "
Nov 02 14:54:44.744327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.744255    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-credential-socket\") pod \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") "
Nov 02 14:54:44.744327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.744308    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-kube-api-access-pz5ws\" (UniqueName: \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-kube-api-access-pz5ws\") pod \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") "
Nov 02 14:54:44.744515 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.744347    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-socket\") pod \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") "
Nov 02 14:54:44.744515 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.744395    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-certs\") pod \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\" (UID: \"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa\") "
Nov 02 14:54:44.744988 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.744895    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-credential-socket" pod "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" (UID: "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:44.745154 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.745091    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-data" pod "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" (UID: "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:44.745407 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.745331    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istiod-ca-cert" pod "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" (UID: "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:54:44.745599 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.745561    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-certs" pod "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" (UID: "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:44.745898 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.745868    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-socket" pod "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" (UID: "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:44.747396 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.747363    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-token" pod "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" (UID: "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:44.748058 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.748014    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-podinfo" pod "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" (UID: "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:54:44.748164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.748091    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-envoy" pod "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" (UID: "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:44.748667 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.748640    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-kube-api-access-pz5ws" pod "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" (UID: "d0db0e10-3bf4-46f5-be72-2ec3687ffcfa"). InnerVolumeSpecName "kube-api-access-pz5ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:44.846450 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.846412    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:44.846450 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.846446    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:44.846450 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.846456    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:44.846450 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.846464    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:44.846887 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.846474    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:44.846887 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.846483    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz5ws\" (UniqueName: \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-kube-api-access-pz5ws\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:44.846887 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.846491    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:44.846887 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.846499    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:44.846887 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:44.846508    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:45.572353 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:45.572314    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s" event={"ID":"d0db0e10-3bf4-46f5-be72-2ec3687ffcfa","Type":"ContainerDied","Data":"96af5885e6420d050a2d2d6c97b319e3a57d5c663b6476d7db61c82fafbaf034"}
Nov 02 14:54:45.572353 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:45.572341    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"
Nov 02 14:54:45.572846 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:45.572393    6284 scope.go:122] "RemoveContainer" containerID="19e67c7b68465aea7d9b7162a8d3672e86788e55534055eb64802662c5ce4a58"
Nov 02 14:54:45.629389 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:45.629349    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"]
Nov 02 14:54:45.638952 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:45.638906    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-45q9s"]
Nov 02 14:54:46.170278 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:46.170235    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" path="/var/lib/kubelet/pods/d0db0e10-3bf4-46f5-be72-2ec3687ffcfa/volumes"
Nov 02 14:54:50.190283 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:54:50.190245    6284 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29c8cf7_2a01_4edc_98be_c7bd46bff630.slice/crio-conmon-88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 14:54:50.190688 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:54:50.190250    6284 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29c8cf7_2a01_4edc_98be_c7bd46bff630.slice/crio-conmon-88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 14:54:50.410304 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.410220    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:50.532752 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.532705    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-envoy\") pod \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") "
Nov 02 14:54:50.532968 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.532771    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-certs\") pod \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") "
Nov 02 14:54:50.532968 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.532902    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-data\") pod \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") "
Nov 02 14:54:50.533083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533004    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-podinfo\") pod \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") "
Nov 02 14:54:50.533173 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533151    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-certs" pod "c29c8cf7-2a01-4edc-98be-c7bd46bff630" (UID: "c29c8cf7-2a01-4edc-98be-c7bd46bff630"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:50.533238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533160    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-credential-socket\") pod \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") "
Nov 02 14:54:50.533323 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533267    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-token\" (UniqueName: \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-token\") pod \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") "
Nov 02 14:54:50.533323 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533299    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istiod-ca-cert\") pod \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") "
Nov 02 14:54:50.533443 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533318    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-data" pod "c29c8cf7-2a01-4edc-98be-c7bd46bff630" (UID: "c29c8cf7-2a01-4edc-98be-c7bd46bff630"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:50.533443 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533336    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-kube-api-access-f4hbf\" (UniqueName: \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-kube-api-access-f4hbf\") pod \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") "
Nov 02 14:54:50.533443 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533361    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-credential-socket" pod "c29c8cf7-2a01-4edc-98be-c7bd46bff630" (UID: "c29c8cf7-2a01-4edc-98be-c7bd46bff630"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:50.533443 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533397    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-socket\") pod \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\" (UID: \"c29c8cf7-2a01-4edc-98be-c7bd46bff630\") "
Nov 02 14:54:50.533670 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533652    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istiod-ca-cert" pod "c29c8cf7-2a01-4edc-98be-c7bd46bff630" (UID: "c29c8cf7-2a01-4edc-98be-c7bd46bff630"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:54:50.533889 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.533863    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-socket" pod "c29c8cf7-2a01-4edc-98be-c7bd46bff630" (UID: "c29c8cf7-2a01-4edc-98be-c7bd46bff630"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:50.534169 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.534148    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:50.534268 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.534174    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:50.534268 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.534189    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:50.534268 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.534200    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:50.534268 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.534215    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:50.535868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.535842    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-envoy" pod "c29c8cf7-2a01-4edc-98be-c7bd46bff630" (UID: "c29c8cf7-2a01-4edc-98be-c7bd46bff630"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:50.535868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.535854    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-token" pod "c29c8cf7-2a01-4edc-98be-c7bd46bff630" (UID: "c29c8cf7-2a01-4edc-98be-c7bd46bff630"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:50.536064 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.535918    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-podinfo" pod "c29c8cf7-2a01-4edc-98be-c7bd46bff630" (UID: "c29c8cf7-2a01-4edc-98be-c7bd46bff630"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:54:50.536175 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.536153    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-kube-api-access-f4hbf" pod "c29c8cf7-2a01-4edc-98be-c7bd46bff630" (UID: "c29c8cf7-2a01-4edc-98be-c7bd46bff630"). InnerVolumeSpecName "kube-api-access-f4hbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:50.608018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.607976    6284 generic.go:356] "Generic (PLEG): container finished" podID="c29c8cf7-2a01-4edc-98be-c7bd46bff630" containerID="88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf" exitCode=0
Nov 02 14:54:50.608243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.608049    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"
Nov 02 14:54:50.608243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.608096    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt" event={"ID":"c29c8cf7-2a01-4edc-98be-c7bd46bff630","Type":"ContainerDied","Data":"88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf"}
Nov 02 14:54:50.608243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.608153    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt" event={"ID":"c29c8cf7-2a01-4edc-98be-c7bd46bff630","Type":"ContainerDied","Data":"e0429111c367a75aadc41499df73d94fc80e94daf9ed1e1b334768fcd5583063"}
Nov 02 14:54:50.608243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.608178    6284 scope.go:122] "RemoveContainer" containerID="88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf"
Nov 02 14:54:50.619624 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.619597    6284 scope.go:122] "RemoveContainer" containerID="88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf"
Nov 02 14:54:50.620002 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:54:50.619977    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf\": container with ID starting with 88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf not found: ID does not exist" containerID="88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf"
Nov 02 14:54:50.620090 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.620012    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf"} err="failed to get container status \"88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf\": rpc error: code = NotFound desc = could not find container \"88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf\": container with ID starting with 88d518d0af1bd1161dccd1d5439a867abbd32ead3c9296884ddd84d715dda8cf not found: ID does not exist"
Nov 02 14:54:50.635467 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.635430    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:50.635467 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.635464    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:50.635467 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.635473    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4hbf\" (UniqueName: \"kubernetes.io/projected/c29c8cf7-2a01-4edc-98be-c7bd46bff630-kube-api-access-f4hbf\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:50.635740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.635482    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c8cf7-2a01-4edc-98be-c7bd46bff630-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:54:50.656233 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.656196    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"]
Nov 02 14:54:50.661643 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:50.661562    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-qg6pt"]
Nov 02 14:54:52.170136 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:54:52.170103    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29c8cf7-2a01-4edc-98be-c7bd46bff630" path="/var/lib/kubelet/pods/c29c8cf7-2a01-4edc-98be-c7bd46bff630/volumes"
Nov 02 14:55:13.895862 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:55:13.895817    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:55:43.896096 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:55:43.896053    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:56:07.049728 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:07.049688    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"]
Nov 02 14:56:13.256955 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:13.256625    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"]
Nov 02 14:56:13.899597 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:56:13.899554    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:56:15.558238 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:15.558198    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"]
Nov 02 14:56:17.755312 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:17.755263    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"]
Nov 02 14:56:22.075818 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.075576    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx" podUID="baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" containerName="main" containerID="cri-o://b6938e2b280e4c9b2b6163ddbabcbec5104682d9d8876cbca4ee006a7dbf1017" gracePeriod=45
Nov 02 14:56:22.230322 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.230279    6284 generic.go:356] "Generic (PLEG): container finished" podID="baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" containerID="b6938e2b280e4c9b2b6163ddbabcbec5104682d9d8876cbca4ee006a7dbf1017" exitCode=0
Nov 02 14:56:22.230598 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.230354    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx" event={"ID":"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd","Type":"ContainerDied","Data":"b6938e2b280e4c9b2b6163ddbabcbec5104682d9d8876cbca4ee006a7dbf1017"}
Nov 02 14:56:22.360960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.360916    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:56:22.523371 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.523328    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-kube-api-access-hk69c\" (UniqueName: \"kubernetes.io/projected/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-kube-api-access-hk69c\") pod \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") "
Nov 02 14:56:22.523588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.523403    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tls-certs\" (UniqueName: \"kubernetes.io/secret/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tls-certs\") pod \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") "
Nov 02 14:56:22.523588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.523434    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-home\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-home\") pod \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") "
Nov 02 14:56:22.523588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.523472    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tmp-dir\") pod \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") "
Nov 02 14:56:22.523588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.523528    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-dshm\") pod \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") "
Nov 02 14:56:22.523588 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.523565    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-model-cache\") pod \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\" (UID: \"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd\") "
Nov 02 14:56:22.523885 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.523765    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-home" pod "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" (UID: "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:22.523885 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.523853    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tmp-dir" pod "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" (UID: "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:22.524005 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.523896    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-model-cache" pod "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" (UID: "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:22.524145 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.524125    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:22.524203 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.524151    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:22.524203 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.524161    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:22.525779 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.525748    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-dshm" pod "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" (UID: "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:22.525930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.525799    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-kube-api-access-hk69c" pod "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" (UID: "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd"). InnerVolumeSpecName "kube-api-access-hk69c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:22.525930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.525834    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tls-certs" pod "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" (UID: "baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:22.625452 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.625356    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:22.625452 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.625392    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk69c\" (UniqueName: \"kubernetes.io/projected/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-kube-api-access-hk69c\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:22.625452 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:22.625404    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:23.239155 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:23.239126    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"
Nov 02 14:56:23.239637 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:23.239158    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx" event={"ID":"baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd","Type":"ContainerDied","Data":"9cb7a19f09fba732d23a3d260227ff922899ecb4d5f5ec46e48e68b908ad5995"}
Nov 02 14:56:23.239637 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:23.239208    6284 scope.go:122] "RemoveContainer" containerID="b6938e2b280e4c9b2b6163ddbabcbec5104682d9d8876cbca4ee006a7dbf1017"
Nov 02 14:56:23.284740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:23.284694    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"]
Nov 02 14:56:23.295572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:23.295529    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-21baa7/rate-test-model-21baa7-kserve-7ffb4d8bc7-5tmcx"]
Nov 02 14:56:24.171726 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:24.171689    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" path="/var/lib/kubelet/pods/baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd/volumes"
Nov 02 14:56:26.606573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:26.606531    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"]
Nov 02 14:56:26.607165 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:26.606833    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj" podUID="92850818-3df8-445f-acda-7c17bf8b0577" containerName="maas-api" containerID="cri-o://65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3" gracePeriod=30
Nov 02 14:56:26.928718 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:26.928689    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:56:27.090538 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.090491    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92850818-3df8-445f-acda-7c17bf8b0577-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92850818-3df8-445f-acda-7c17bf8b0577-openshift-service-ca\") pod \"92850818-3df8-445f-acda-7c17bf8b0577\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") "
Nov 02 14:56:27.090757 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.090658    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-metrics-certs\") pod \"92850818-3df8-445f-acda-7c17bf8b0577\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") "
Nov 02 14:56:27.091018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.090926    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-maas-api-tls\") pod \"92850818-3df8-445f-acda-7c17bf8b0577\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") "
Nov 02 14:56:27.091018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.090985    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92850818-3df8-445f-acda-7c17bf8b0577-openshift-service-ca" pod "92850818-3df8-445f-acda-7c17bf8b0577" (UID: "92850818-3df8-445f-acda-7c17bf8b0577"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:27.091356 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.091048    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92850818-3df8-445f-acda-7c17bf8b0577-kube-api-access-km5vn\" (UniqueName: \"kubernetes.io/projected/92850818-3df8-445f-acda-7c17bf8b0577-kube-api-access-km5vn\") pod \"92850818-3df8-445f-acda-7c17bf8b0577\" (UID: \"92850818-3df8-445f-acda-7c17bf8b0577\") "
Nov 02 14:56:27.092518 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.092485    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92850818-3df8-445f-acda-7c17bf8b0577-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:27.093385 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.093356    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-metrics-certs" pod "92850818-3df8-445f-acda-7c17bf8b0577" (UID: "92850818-3df8-445f-acda-7c17bf8b0577"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:27.093536 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.093512    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-maas-api-tls" pod "92850818-3df8-445f-acda-7c17bf8b0577" (UID: "92850818-3df8-445f-acda-7c17bf8b0577"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:27.094642 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.094618    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92850818-3df8-445f-acda-7c17bf8b0577-kube-api-access-km5vn" pod "92850818-3df8-445f-acda-7c17bf8b0577" (UID: "92850818-3df8-445f-acda-7c17bf8b0577"). InnerVolumeSpecName "kube-api-access-km5vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:27.194057 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.193889    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:27.194057 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.193990    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km5vn\" (UniqueName: \"kubernetes.io/projected/92850818-3df8-445f-acda-7c17bf8b0577-kube-api-access-km5vn\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:27.194057 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.194008    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92850818-3df8-445f-acda-7c17bf8b0577-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:27.268251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.268207    6284 generic.go:356] "Generic (PLEG): container finished" podID="92850818-3df8-445f-acda-7c17bf8b0577" containerID="65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3" exitCode=0
Nov 02 14:56:27.268251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.268256    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj" event={"ID":"92850818-3df8-445f-acda-7c17bf8b0577","Type":"ContainerDied","Data":"65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3"}
Nov 02 14:56:27.268507 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.268259    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"
Nov 02 14:56:27.268507 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.268300    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj" event={"ID":"92850818-3df8-445f-acda-7c17bf8b0577","Type":"ContainerDied","Data":"935a276adc57cf9c13daee729c8fc09acc0a7654a3fb44ad43be81b37489d08b"}
Nov 02 14:56:27.268507 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.268323    6284 scope.go:122] "RemoveContainer" containerID="65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3"
Nov 02 14:56:27.280159 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.280137    6284 scope.go:122] "RemoveContainer" containerID="65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3"
Nov 02 14:56:27.280516 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:56:27.280493    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3\": container with ID starting with 65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3 not found: ID does not exist" containerID="65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3"
Nov 02 14:56:27.280565 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.280531    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3"} err="failed to get container status \"65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3\": rpc error: code = NotFound desc = could not find container \"65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3\": container with ID starting with 65ddf12b0a2bfb3dab75233f5384a33ec0830cf4a1912fb0a1eeb4596556ebb3 not found: ID does not exist"
Nov 02 14:56:27.317205 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.317166    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"]
Nov 02 14:56:27.328792 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:27.328744    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ce0f55-cbddbff96-vd5gj"]
Nov 02 14:56:28.170123 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.170088    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92850818-3df8-445f-acda-7c17bf8b0577" path="/var/lib/kubelet/pods/92850818-3df8-445f-acda-7c17bf8b0577/volumes"
Nov 02 14:56:28.281475 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.281430    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb" podUID="1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" containerName="main" containerID="cri-o://36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d" gracePeriod=45
Nov 02 14:56:28.563026 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.562993    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:56:28.606056 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.606023    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-kube-api-access-qms6x\" (UniqueName: \"kubernetes.io/projected/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-kube-api-access-qms6x\") pod \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") "
Nov 02 14:56:28.606273 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.606138    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tmp-dir\") pod \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") "
Nov 02 14:56:28.606273 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.606211    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-home\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-home\") pod \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") "
Nov 02 14:56:28.606273 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.606261    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-model-cache\") pod \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") "
Nov 02 14:56:28.606456 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.606296    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tls-certs\" (UniqueName: \"kubernetes.io/secret/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tls-certs\") pod \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") "
Nov 02 14:56:28.606456 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.606382    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-dshm\") pod \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\" (UID: \"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0\") "
Nov 02 14:56:28.606573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.606504    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tmp-dir" pod "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" (UID: "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:28.606805 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.606766    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:28.607054 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.607021    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-model-cache" pod "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" (UID: "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:28.607242 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.607224    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-home" pod "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" (UID: "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:28.608657 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.608629    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-dshm" pod "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" (UID: "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:28.608800 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.608781    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-kube-api-access-qms6x" pod "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" (UID: "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0"). InnerVolumeSpecName "kube-api-access-qms6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:28.608906 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.608891    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tls-certs" pod "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" (UID: "1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:28.708265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.708173    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qms6x\" (UniqueName: \"kubernetes.io/projected/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-kube-api-access-qms6x\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:28.708265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.708208    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:28.708265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.708220    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:28.708265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.708228    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:28.708265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:28.708236    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:29.283580 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:29.283545    6284 generic.go:356] "Generic (PLEG): container finished" podID="1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" containerID="36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d" exitCode=0
Nov 02 14:56:29.284073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:29.283596    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"
Nov 02 14:56:29.284073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:29.283632    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb" event={"ID":"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0","Type":"ContainerDied","Data":"36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d"}
Nov 02 14:56:29.284073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:29.283681    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb" event={"ID":"1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0","Type":"ContainerDied","Data":"a39f686871cf72393be55cda71bcfc39cca80b249a0fbd93571206e300e58199"}
Nov 02 14:56:29.284073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:29.283707    6284 scope.go:122] "RemoveContainer" containerID="36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d"
Nov 02 14:56:29.294558 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:29.294535    6284 scope.go:122] "RemoveContainer" containerID="36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d"
Nov 02 14:56:29.294977 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:56:29.294923    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d\": container with ID starting with 36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d not found: ID does not exist" containerID="36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d"
Nov 02 14:56:29.295024 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:29.294991    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d"} err="failed to get container status \"36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d\": rpc error: code = NotFound desc = could not find container \"36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d\": container with ID starting with 36f1c249bdeaa14287860bffb0a10812295b126dadba47548a109a3ced54682d not found: ID does not exist"
Nov 02 14:56:29.334428 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:29.334385    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"]
Nov 02 14:56:29.343818 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:29.343778    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2cddf2/rate-test-model-2cddf2-kserve-64d65cfc8f-tz9wb"]
Nov 02 14:56:30.170569 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.170531    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" path="/var/lib/kubelet/pods/1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0/volumes"
Nov 02 14:56:30.590977 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.590914    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58" podUID="261ca16d-584d-4b15-b34c-0a54eb966013" containerName="main" containerID="cri-o://762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025" gracePeriod=45
Nov 02 14:56:30.636625 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:56:30.636579    6284 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod261ca16d_584d_4b15_b34c_0a54eb966013.slice/crio-762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod261ca16d_584d_4b15_b34c_0a54eb966013.slice/crio-conmon-762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 14:56:30.875658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.875628    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:56:30.935714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.935673    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-tmp-dir\") pod \"261ca16d-584d-4b15-b34c-0a54eb966013\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") "
Nov 02 14:56:30.935959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.935798    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-model-cache\") pod \"261ca16d-584d-4b15-b34c-0a54eb966013\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") "
Nov 02 14:56:30.935959 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.935871    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-dshm\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-dshm\") pod \"261ca16d-584d-4b15-b34c-0a54eb966013\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") "
Nov 02 14:56:30.936100 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.936017    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-tmp-dir" pod "261ca16d-584d-4b15-b34c-0a54eb966013" (UID: "261ca16d-584d-4b15-b34c-0a54eb966013"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:30.936229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.936205    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-model-cache" pod "261ca16d-584d-4b15-b34c-0a54eb966013" (UID: "261ca16d-584d-4b15-b34c-0a54eb966013"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:30.936417 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.936396    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-home\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-home\") pod \"261ca16d-584d-4b15-b34c-0a54eb966013\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") "
Nov 02 14:56:30.936465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.936443    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261ca16d-584d-4b15-b34c-0a54eb966013-kube-api-access-sfrgm\" (UniqueName: \"kubernetes.io/projected/261ca16d-584d-4b15-b34c-0a54eb966013-kube-api-access-sfrgm\") pod \"261ca16d-584d-4b15-b34c-0a54eb966013\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") "
Nov 02 14:56:30.936521 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.936478    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261ca16d-584d-4b15-b34c-0a54eb966013-tls-certs\" (UniqueName: \"kubernetes.io/secret/261ca16d-584d-4b15-b34c-0a54eb966013-tls-certs\") pod \"261ca16d-584d-4b15-b34c-0a54eb966013\" (UID: \"261ca16d-584d-4b15-b34c-0a54eb966013\") "
Nov 02 14:56:30.936689 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.936652    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-home" pod "261ca16d-584d-4b15-b34c-0a54eb966013" (UID: "261ca16d-584d-4b15-b34c-0a54eb966013"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:30.937141 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.937119    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:30.937244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.937147    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:30.937244 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.937161    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:30.939356 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.939330    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261ca16d-584d-4b15-b34c-0a54eb966013-tls-certs" pod "261ca16d-584d-4b15-b34c-0a54eb966013" (UID: "261ca16d-584d-4b15-b34c-0a54eb966013"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:30.939448 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.939354    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-dshm" pod "261ca16d-584d-4b15-b34c-0a54eb966013" (UID: "261ca16d-584d-4b15-b34c-0a54eb966013"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:30.939511 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:30.939491    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261ca16d-584d-4b15-b34c-0a54eb966013-kube-api-access-sfrgm" pod "261ca16d-584d-4b15-b34c-0a54eb966013" (UID: "261ca16d-584d-4b15-b34c-0a54eb966013"). InnerVolumeSpecName "kube-api-access-sfrgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:31.038576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.038531    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261ca16d-584d-4b15-b34c-0a54eb966013-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:31.038576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.038568    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfrgm\" (UniqueName: \"kubernetes.io/projected/261ca16d-584d-4b15-b34c-0a54eb966013-kube-api-access-sfrgm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:31.038576 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.038579    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261ca16d-584d-4b15-b34c-0a54eb966013-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:31.298894 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.298853    6284 generic.go:356] "Generic (PLEG): container finished" podID="261ca16d-584d-4b15-b34c-0a54eb966013" containerID="762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025" exitCode=0
Nov 02 14:56:31.299111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.298906    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"
Nov 02 14:56:31.299111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.298953    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58" event={"ID":"261ca16d-584d-4b15-b34c-0a54eb966013","Type":"ContainerDied","Data":"762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025"}
Nov 02 14:56:31.299111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.299011    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58" event={"ID":"261ca16d-584d-4b15-b34c-0a54eb966013","Type":"ContainerDied","Data":"32a9ed6bbd1752cfaf7de961ef7c87b6ce24ea8b7da8ead2fb7bf059676475b8"}
Nov 02 14:56:31.299111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.299034    6284 scope.go:122] "RemoveContainer" containerID="762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025"
Nov 02 14:56:31.310168 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.310144    6284 scope.go:122] "RemoveContainer" containerID="762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025"
Nov 02 14:56:31.310517 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:56:31.310491    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025\": container with ID starting with 762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025 not found: ID does not exist" containerID="762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025"
Nov 02 14:56:31.310571 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.310527    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025"} err="failed to get container status \"762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025\": rpc error: code = NotFound desc = could not find container \"762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025\": container with ID starting with 762d0b0aae2e6cf1f7fd0f197cb0cf4bafc486d3560efb28416364cd24944025 not found: ID does not exist"
Nov 02 14:56:31.343158 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.343114    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"]
Nov 02 14:56:31.347267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:31.347232    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4c2811/rate-test-model-4c2811-kserve-78986bc477-74p58"]
Nov 02 14:56:32.172469 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:32.172425    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261ca16d-584d-4b15-b34c-0a54eb966013" path="/var/lib/kubelet/pods/261ca16d-584d-4b15-b34c-0a54eb966013/volumes"
Nov 02 14:56:32.782034 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:32.781960    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr" podUID="adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" containerName="main" containerID="cri-o://42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb" gracePeriod=45
Nov 02 14:56:32.838696 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:32.838611    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"]
Nov 02 14:56:32.839073 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:32.839035    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" podUID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" containerName="istio-proxy" containerID="cri-o://0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e" gracePeriod=30
Nov 02 14:56:32.932566 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:32.932518    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr" podUID="adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": dial tcp 10.134.0.167:8000: connect: connection refused"
Nov 02 14:56:33.078665 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.078629    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:56:33.169303 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.169260    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tmp-dir\") pod \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") "
Nov 02 14:56:33.169504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.169346    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-home\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-home\") pod \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") "
Nov 02 14:56:33.169504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.169398    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-kube-api-access-g7ndx\" (UniqueName: \"kubernetes.io/projected/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-kube-api-access-g7ndx\") pod \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") "
Nov 02 14:56:33.169504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.169423    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tls-certs\" (UniqueName: \"kubernetes.io/secret/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tls-certs\") pod \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") "
Nov 02 14:56:33.169504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.169492    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-dshm\") pod \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") "
Nov 02 14:56:33.169712 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.169512    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-model-cache\") pod \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\" (UID: \"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e\") "
Nov 02 14:56:33.169712 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.169626    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tmp-dir" pod "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" (UID: "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:33.169821 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.169769    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-home" pod "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" (UID: "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:33.169889 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.169867    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-model-cache" pod "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" (UID: "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:33.170142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.170123    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:33.170142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.170143    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:33.170142 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.170151    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:33.171777 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.171744    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-dshm" pod "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" (UID: "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:33.171893 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.171789    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-kube-api-access-g7ndx" pod "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" (UID: "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e"). InnerVolumeSpecName "kube-api-access-g7ndx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:33.171893 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.171806    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tls-certs" pod "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" (UID: "adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:33.271208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.271164    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:33.271208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.271198    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7ndx\" (UniqueName: \"kubernetes.io/projected/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-kube-api-access-g7ndx\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:33.271208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.271208    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:33.320582 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.320483    6284 generic.go:356] "Generic (PLEG): container finished" podID="adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" containerID="42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb" exitCode=0
Nov 02 14:56:33.320582 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.320531    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"
Nov 02 14:56:33.320799 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.320572    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr" event={"ID":"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e","Type":"ContainerDied","Data":"42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb"}
Nov 02 14:56:33.320799 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.320620    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr" event={"ID":"adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e","Type":"ContainerDied","Data":"029dd9d7d171b209d09a1c6b274aa218531562827b23ed15654f558c28bc08f6"}
Nov 02 14:56:33.320799 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.320640    6284 scope.go:122] "RemoveContainer" containerID="42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb"
Nov 02 14:56:33.332125 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.332100    6284 scope.go:122] "RemoveContainer" containerID="42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb"
Nov 02 14:56:33.332496 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:56:33.332471    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb\": container with ID starting with 42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb not found: ID does not exist" containerID="42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb"
Nov 02 14:56:33.332557 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.332507    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb"} err="failed to get container status \"42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb\": rpc error: code = NotFound desc = could not find container \"42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb\": container with ID starting with 42b2a9baac84823c49a51936cabc8ebdf8f7ba822f612f0430032f00afbd96eb not found: ID does not exist"
Nov 02 14:56:33.365408 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.365360    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"]
Nov 02 14:56:33.369808 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:33.369766    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0d1ca7/rate-test-model-0d1ca7-kserve-b684dc889-jwktr"]
Nov 02 14:56:34.169839 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:34.169798    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" path="/var/lib/kubelet/pods/adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e/volumes"
Nov 02 14:56:38.127857 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.127790    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:56:38.232604 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.232563    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-credential-socket\") pod \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") "
Nov 02 14:56:38.232788 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.232648    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-data\") pod \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") "
Nov 02 14:56:38.232788 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.232688    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-token\") pod \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") "
Nov 02 14:56:38.232788 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.232741    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-kube-api-access-h6sqt\" (UniqueName: \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-kube-api-access-h6sqt\") pod \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") "
Nov 02 14:56:38.232929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.232799    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-certs\") pod \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") "
Nov 02 14:56:38.232929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.232851    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-socket\") pod \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") "
Nov 02 14:56:38.232929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.232879    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-envoy\") pod \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") "
Nov 02 14:56:38.232929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.232909    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istiod-ca-cert\") pod \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") "
Nov 02 14:56:38.233163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.232908    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-credential-socket" pod "b8fc299a-3d6d-4da3-94de-a974f6e3afc1" (UID: "b8fc299a-3d6d-4da3-94de-a974f6e3afc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:38.233163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.232978    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-data" pod "b8fc299a-3d6d-4da3-94de-a974f6e3afc1" (UID: "b8fc299a-3d6d-4da3-94de-a974f6e3afc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:38.233163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.233068    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-podinfo\") pod \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\" (UID: \"b8fc299a-3d6d-4da3-94de-a974f6e3afc1\") "
Nov 02 14:56:38.233318 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.233224    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-socket" pod "b8fc299a-3d6d-4da3-94de-a974f6e3afc1" (UID: "b8fc299a-3d6d-4da3-94de-a974f6e3afc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:38.233318 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.233240    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-certs" pod "b8fc299a-3d6d-4da3-94de-a974f6e3afc1" (UID: "b8fc299a-3d6d-4da3-94de-a974f6e3afc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:38.233418 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.233317    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istiod-ca-cert" pod "b8fc299a-3d6d-4da3-94de-a974f6e3afc1" (UID: "b8fc299a-3d6d-4da3-94de-a974f6e3afc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:38.233892 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.233867    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:38.234127 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.234099    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:38.234127 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.234127    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:38.234301 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.234142    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:38.234301 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.234155    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:38.235249 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.235222    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-token" pod "b8fc299a-3d6d-4da3-94de-a974f6e3afc1" (UID: "b8fc299a-3d6d-4da3-94de-a974f6e3afc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:38.235388 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.235267    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-kube-api-access-h6sqt" pod "b8fc299a-3d6d-4da3-94de-a974f6e3afc1" (UID: "b8fc299a-3d6d-4da3-94de-a974f6e3afc1"). InnerVolumeSpecName "kube-api-access-h6sqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:38.235388 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.235362    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-envoy" pod "b8fc299a-3d6d-4da3-94de-a974f6e3afc1" (UID: "b8fc299a-3d6d-4da3-94de-a974f6e3afc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:38.235465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.235401    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-podinfo" pod "b8fc299a-3d6d-4da3-94de-a974f6e3afc1" (UID: "b8fc299a-3d6d-4da3-94de-a974f6e3afc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:56:38.335558 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.335510    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:38.335558 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.335554    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:38.335558 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.335565    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:38.335823 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.335576    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6sqt\" (UniqueName: \"kubernetes.io/projected/b8fc299a-3d6d-4da3-94de-a974f6e3afc1-kube-api-access-h6sqt\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:38.355924 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.355885    6284 generic.go:356] "Generic (PLEG): container finished" podID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" containerID="0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e" exitCode=0
Nov 02 14:56:38.356131 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.355975    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"
Nov 02 14:56:38.356131 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.355992    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" event={"ID":"b8fc299a-3d6d-4da3-94de-a974f6e3afc1","Type":"ContainerDied","Data":"0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e"}
Nov 02 14:56:38.356131 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.356048    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj" event={"ID":"b8fc299a-3d6d-4da3-94de-a974f6e3afc1","Type":"ContainerDied","Data":"ea467b9cd23760a2de2b85f61a6d1b19d3e475c326ab5e0829c2a40f445bbffb"}
Nov 02 14:56:38.356131 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.356073    6284 scope.go:122] "RemoveContainer" containerID="0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e"
Nov 02 14:56:38.367683 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.367660    6284 scope.go:122] "RemoveContainer" containerID="0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e"
Nov 02 14:56:38.368066 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:56:38.368038    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e\": container with ID starting with 0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e not found: ID does not exist" containerID="0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e"
Nov 02 14:56:38.368155 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.368074    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e"} err="failed to get container status \"0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e\": rpc error: code = NotFound desc = could not find container \"0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e\": container with ID starting with 0132eefc4d3e5c500740121f88bbff3bca4d736ae55f817aac7931b88330d20e not found: ID does not exist"
Nov 02 14:56:38.401819 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.401728    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"]
Nov 02 14:56:38.404702 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:38.404666    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-gkrxj"]
Nov 02 14:56:39.339713 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:39.339669    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"]
Nov 02 14:56:40.170530 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:40.170488    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" path="/var/lib/kubelet/pods/b8fc299a-3d6d-4da3-94de-a974f6e3afc1/volumes"
Nov 02 14:56:43.896306 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:56:43.896260    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:56:44.340526 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:44.340467    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t" podUID="91688d53-63bb-4a17-91c8-8e9d5f722eb4" containerName="payload-pre-processing" containerID="cri-o://49c16cc5b16dca102c1883a34b6afb34ec5c82ee8e46f01bb70ebdad8f83b478" gracePeriod=25
Nov 02 14:56:47.472078 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:47.472036    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t" podUID="91688d53-63bb-4a17-91c8-8e9d5f722eb4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:56:53.704956 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:53.698346    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"]
Nov 02 14:56:56.998160 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:56.998119    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"]
Nov 02 14:56:56.998714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:56.998390    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd" podUID="fc70a0fe-2fe3-41be-b674-f96892b5ae58" containerName="maas-api" containerID="cri-o://d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b" gracePeriod=30
Nov 02 14:56:57.324814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.324776    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:56:57.365532 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.365487    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc70a0fe-2fe3-41be-b674-f96892b5ae58-kube-api-access-qtxlb\" (UniqueName: \"kubernetes.io/projected/fc70a0fe-2fe3-41be-b674-f96892b5ae58-kube-api-access-qtxlb\") pod \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") "
Nov 02 14:56:57.365787 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.365564    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-metrics-certs\") pod \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") "
Nov 02 14:56:57.365787 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.365734    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-maas-api-tls\") pod \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") "
Nov 02 14:56:57.365929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.365819    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc70a0fe-2fe3-41be-b674-f96892b5ae58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc70a0fe-2fe3-41be-b674-f96892b5ae58-openshift-service-ca\") pod \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\" (UID: \"fc70a0fe-2fe3-41be-b674-f96892b5ae58\") "
Nov 02 14:56:57.366317 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.366282    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc70a0fe-2fe3-41be-b674-f96892b5ae58-openshift-service-ca" pod "fc70a0fe-2fe3-41be-b674-f96892b5ae58" (UID: "fc70a0fe-2fe3-41be-b674-f96892b5ae58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:57.368127 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.368092    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-metrics-certs" pod "fc70a0fe-2fe3-41be-b674-f96892b5ae58" (UID: "fc70a0fe-2fe3-41be-b674-f96892b5ae58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.368288 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.368238    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-maas-api-tls" pod "fc70a0fe-2fe3-41be-b674-f96892b5ae58" (UID: "fc70a0fe-2fe3-41be-b674-f96892b5ae58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.368363 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.368303    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc70a0fe-2fe3-41be-b674-f96892b5ae58-kube-api-access-qtxlb" pod "fc70a0fe-2fe3-41be-b674-f96892b5ae58" (UID: "fc70a0fe-2fe3-41be-b674-f96892b5ae58"). InnerVolumeSpecName "kube-api-access-qtxlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:57.467268 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.467228    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.467268 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.467264    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc70a0fe-2fe3-41be-b674-f96892b5ae58-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.467268 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.467277    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtxlb\" (UniqueName: \"kubernetes.io/projected/fc70a0fe-2fe3-41be-b674-f96892b5ae58-kube-api-access-qtxlb\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.467580 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.467290    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc70a0fe-2fe3-41be-b674-f96892b5ae58-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.472455 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.472407    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t" podUID="91688d53-63bb-4a17-91c8-8e9d5f722eb4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:56:57.479414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.479377    6284 generic.go:356] "Generic (PLEG): container finished" podID="fc70a0fe-2fe3-41be-b674-f96892b5ae58" containerID="d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b" exitCode=0
Nov 02 14:56:57.479595 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.479430    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"
Nov 02 14:56:57.479595 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.479436    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd" event={"ID":"fc70a0fe-2fe3-41be-b674-f96892b5ae58","Type":"ContainerDied","Data":"d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b"}
Nov 02 14:56:57.479595 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.479494    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd" event={"ID":"fc70a0fe-2fe3-41be-b674-f96892b5ae58","Type":"ContainerDied","Data":"e48e081cf01c0339a571e439959d180f36bf935a4c59f27d4369355228a98659"}
Nov 02 14:56:57.479595 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.479526    6284 scope.go:122] "RemoveContainer" containerID="d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b"
Nov 02 14:56:57.493533 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.493503    6284 scope.go:122] "RemoveContainer" containerID="d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b"
Nov 02 14:56:57.493856 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:56:57.493832    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b\": container with ID starting with d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b not found: ID does not exist" containerID="d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b"
Nov 02 14:56:57.493960 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.493868    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b"} err="failed to get container status \"d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b\": rpc error: code = NotFound desc = could not find container \"d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b\": container with ID starting with d2d188fc2eb41693212d541261e41e0e8d008d36aabde2b715766d84ba17177b not found: ID does not exist"
Nov 02 14:56:57.525248 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.525206    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"]
Nov 02 14:56:57.528164 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:57.528131    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-66bc6d-bb7445cf4-ghnmd"]
Nov 02 14:56:58.169346 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:56:58.169306    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc70a0fe-2fe3-41be-b674-f96892b5ae58" path="/var/lib/kubelet/pods/fc70a0fe-2fe3-41be-b674-f96892b5ae58/volumes"
Nov 02 14:57:02.699230 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:02.699190    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"]
Nov 02 14:57:02.699731 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:02.699482    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl" podUID="6ef217bd-b6ff-4c32-9591-8584e524bcd5" containerName="maas-api" containerID="cri-o://7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965" gracePeriod=30
Nov 02 14:57:02.993819 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:02.993789    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:57:03.138348 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.138317    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-maas-api-tls\") pod \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") "
Nov 02 14:57:03.138572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.138412    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef217bd-b6ff-4c32-9591-8584e524bcd5-kube-api-access-crgg9\" (UniqueName: \"kubernetes.io/projected/6ef217bd-b6ff-4c32-9591-8584e524bcd5-kube-api-access-crgg9\") pod \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") "
Nov 02 14:57:03.138572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.138442    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef217bd-b6ff-4c32-9591-8584e524bcd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef217bd-b6ff-4c32-9591-8584e524bcd5-openshift-service-ca\") pod \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") "
Nov 02 14:57:03.138572 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.138483    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-metrics-certs\") pod \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\" (UID: \"6ef217bd-b6ff-4c32-9591-8584e524bcd5\") "
Nov 02 14:57:03.138903 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.138876    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef217bd-b6ff-4c32-9591-8584e524bcd5-openshift-service-ca" pod "6ef217bd-b6ff-4c32-9591-8584e524bcd5" (UID: "6ef217bd-b6ff-4c32-9591-8584e524bcd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:03.139083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.139063    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef217bd-b6ff-4c32-9591-8584e524bcd5-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:03.140804 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.140773    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-maas-api-tls" pod "6ef217bd-b6ff-4c32-9591-8584e524bcd5" (UID: "6ef217bd-b6ff-4c32-9591-8584e524bcd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:03.140804 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.140795    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-metrics-certs" pod "6ef217bd-b6ff-4c32-9591-8584e524bcd5" (UID: "6ef217bd-b6ff-4c32-9591-8584e524bcd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:03.141023 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.141005    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef217bd-b6ff-4c32-9591-8584e524bcd5-kube-api-access-crgg9" pod "6ef217bd-b6ff-4c32-9591-8584e524bcd5" (UID: "6ef217bd-b6ff-4c32-9591-8584e524bcd5"). InnerVolumeSpecName "kube-api-access-crgg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:03.240027 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.239980    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:03.240027 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.240023    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crgg9\" (UniqueName: \"kubernetes.io/projected/6ef217bd-b6ff-4c32-9591-8584e524bcd5-kube-api-access-crgg9\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:03.240304 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.240039    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef217bd-b6ff-4c32-9591-8584e524bcd5-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:03.520634 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.520539    6284 generic.go:356] "Generic (PLEG): container finished" podID="6ef217bd-b6ff-4c32-9591-8584e524bcd5" containerID="7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965" exitCode=0
Nov 02 14:57:03.520634 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.520588    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl" event={"ID":"6ef217bd-b6ff-4c32-9591-8584e524bcd5","Type":"ContainerDied","Data":"7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965"}
Nov 02 14:57:03.520634 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.520603    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"
Nov 02 14:57:03.520931 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.520639    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl" event={"ID":"6ef217bd-b6ff-4c32-9591-8584e524bcd5","Type":"ContainerDied","Data":"4592fee834004a59108efc86800efaf03996766b6ab28c1cb6f5f6b9357c9d0b"}
Nov 02 14:57:03.520931 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.520665    6284 scope.go:122] "RemoveContainer" containerID="7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965"
Nov 02 14:57:03.531800 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.531772    6284 scope.go:122] "RemoveContainer" containerID="7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965"
Nov 02 14:57:03.532230 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:57:03.532204    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965\": container with ID starting with 7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965 not found: ID does not exist" containerID="7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965"
Nov 02 14:57:03.532298 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.532246    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965"} err="failed to get container status \"7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965\": rpc error: code = NotFound desc = could not find container \"7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965\": container with ID starting with 7d0fadb731df8f047555cc555dc6d7bb3d36fb34a54b196449610fce04e3f965 not found: ID does not exist"
Nov 02 14:57:03.565218 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.565181    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"]
Nov 02 14:57:03.575146 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:03.575108    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5f98a6-59f554d657-phknl"]
Nov 02 14:57:04.169243 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:04.169203    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef217bd-b6ff-4c32-9591-8584e524bcd5" path="/var/lib/kubelet/pods/6ef217bd-b6ff-4c32-9591-8584e524bcd5/volumes"
Nov 02 14:57:04.536425 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:04.536379    6284 generic.go:356] "Generic (PLEG): container finished" podID="91688d53-63bb-4a17-91c8-8e9d5f722eb4" containerID="49c16cc5b16dca102c1883a34b6afb34ec5c82ee8e46f01bb70ebdad8f83b478" exitCode=0
Nov 02 14:57:04.536592 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:04.536456    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t" event={"ID":"91688d53-63bb-4a17-91c8-8e9d5f722eb4","Type":"ContainerDied","Data":"49c16cc5b16dca102c1883a34b6afb34ec5c82ee8e46f01bb70ebdad8f83b478"}
Nov 02 14:57:04.604745 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:04.604715    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:57:04.657631 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:04.657594    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91688d53-63bb-4a17-91c8-8e9d5f722eb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91688d53-63bb-4a17-91c8-8e9d5f722eb4-plugins-config-volume\") pod \"91688d53-63bb-4a17-91c8-8e9d5f722eb4\" (UID: \"91688d53-63bb-4a17-91c8-8e9d5f722eb4\") "
Nov 02 14:57:04.658538 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:04.657683    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91688d53-63bb-4a17-91c8-8e9d5f722eb4-kube-api-access-t8k2g\" (UniqueName: \"kubernetes.io/projected/91688d53-63bb-4a17-91c8-8e9d5f722eb4-kube-api-access-t8k2g\") pod \"91688d53-63bb-4a17-91c8-8e9d5f722eb4\" (UID: \"91688d53-63bb-4a17-91c8-8e9d5f722eb4\") "
Nov 02 14:57:04.658538 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:04.658512    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91688d53-63bb-4a17-91c8-8e9d5f722eb4-plugins-config-volume" pod "91688d53-63bb-4a17-91c8-8e9d5f722eb4" (UID: "91688d53-63bb-4a17-91c8-8e9d5f722eb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:04.660639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:04.660605    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91688d53-63bb-4a17-91c8-8e9d5f722eb4-kube-api-access-t8k2g" pod "91688d53-63bb-4a17-91c8-8e9d5f722eb4" (UID: "91688d53-63bb-4a17-91c8-8e9d5f722eb4"). InnerVolumeSpecName "kube-api-access-t8k2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:04.759499 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:04.759455    6284 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91688d53-63bb-4a17-91c8-8e9d5f722eb4-plugins-config-volume\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:04.759499 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:04.759499    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8k2g\" (UniqueName: \"kubernetes.io/projected/91688d53-63bb-4a17-91c8-8e9d5f722eb4-kube-api-access-t8k2g\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:05.543706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:05.543670    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"
Nov 02 14:57:05.544177 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:05.543670    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t" event={"ID":"91688d53-63bb-4a17-91c8-8e9d5f722eb4","Type":"ContainerDied","Data":"ddeeb15b03b31ffe3f577dcfc4593bce553619727e306e189a0b91a4b1fbd315"}
Nov 02 14:57:05.544177 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:05.543820    6284 scope.go:122] "RemoveContainer" containerID="49c16cc5b16dca102c1883a34b6afb34ec5c82ee8e46f01bb70ebdad8f83b478"
Nov 02 14:57:05.590516 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:05.590473    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"]
Nov 02 14:57:05.597569 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:05.597525    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-66bc6d-8bdcd98b7-7x88t"]
Nov 02 14:57:06.169390 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:06.169349    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91688d53-63bb-4a17-91c8-8e9d5f722eb4" path="/var/lib/kubelet/pods/91688d53-63bb-4a17-91c8-8e9d5f722eb4/volumes"
Nov 02 14:57:08.722975 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:08.722911    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c" podUID="0eb75de2-3838-4b3e-a5ab-85e2426628e7" containerName="main" containerID="cri-o://d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea" gracePeriod=45
Nov 02 14:57:08.992386 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:08.992356    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:57:09.112529 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.112490    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tmp-dir\") pod \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") "
Nov 02 14:57:09.112529 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.112537    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-dshm\") pod \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") "
Nov 02 14:57:09.112780 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.112615    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb75de2-3838-4b3e-a5ab-85e2426628e7-kube-api-access-jnn7q\" (UniqueName: \"kubernetes.io/projected/0eb75de2-3838-4b3e-a5ab-85e2426628e7-kube-api-access-jnn7q\") pod \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") "
Nov 02 14:57:09.112780 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.112734    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-model-cache\") pod \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") "
Nov 02 14:57:09.112890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.112822    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-home\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-home\") pod \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") "
Nov 02 14:57:09.112890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.112838    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tmp-dir" pod "0eb75de2-3838-4b3e-a5ab-85e2426628e7" (UID: "0eb75de2-3838-4b3e-a5ab-85e2426628e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:09.112890 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.112852    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tls-certs\") pod \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\" (UID: \"0eb75de2-3838-4b3e-a5ab-85e2426628e7\") "
Nov 02 14:57:09.113105 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.113079    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-model-cache" pod "0eb75de2-3838-4b3e-a5ab-85e2426628e7" (UID: "0eb75de2-3838-4b3e-a5ab-85e2426628e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:09.113174 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.113127    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-home" pod "0eb75de2-3838-4b3e-a5ab-85e2426628e7" (UID: "0eb75de2-3838-4b3e-a5ab-85e2426628e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:09.113372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.113349    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:09.113372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.113369    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:09.113372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.113378    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:09.115186 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.115156    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-dshm" pod "0eb75de2-3838-4b3e-a5ab-85e2426628e7" (UID: "0eb75de2-3838-4b3e-a5ab-85e2426628e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:09.115300 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.115212    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tls-certs" pod "0eb75de2-3838-4b3e-a5ab-85e2426628e7" (UID: "0eb75de2-3838-4b3e-a5ab-85e2426628e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:09.115300 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.115231    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb75de2-3838-4b3e-a5ab-85e2426628e7-kube-api-access-jnn7q" pod "0eb75de2-3838-4b3e-a5ab-85e2426628e7" (UID: "0eb75de2-3838-4b3e-a5ab-85e2426628e7"). InnerVolumeSpecName "kube-api-access-jnn7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:09.214219 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.214175    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb75de2-3838-4b3e-a5ab-85e2426628e7-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:09.214219 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.214212    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb75de2-3838-4b3e-a5ab-85e2426628e7-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:09.214219 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.214223    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnn7q\" (UniqueName: \"kubernetes.io/projected/0eb75de2-3838-4b3e-a5ab-85e2426628e7-kube-api-access-jnn7q\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:09.574016 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.573980    6284 generic.go:356] "Generic (PLEG): container finished" podID="0eb75de2-3838-4b3e-a5ab-85e2426628e7" containerID="d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea" exitCode=0
Nov 02 14:57:09.574206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.574027    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"
Nov 02 14:57:09.574206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.574063    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c" event={"ID":"0eb75de2-3838-4b3e-a5ab-85e2426628e7","Type":"ContainerDied","Data":"d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea"}
Nov 02 14:57:09.574206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.574114    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c" event={"ID":"0eb75de2-3838-4b3e-a5ab-85e2426628e7","Type":"ContainerDied","Data":"3363b86251c6f51211e70fdae4bd4277e7b9f368493e4945260f1378c41155b9"}
Nov 02 14:57:09.574206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.574132    6284 scope.go:122] "RemoveContainer" containerID="d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea"
Nov 02 14:57:09.584644 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.584622    6284 scope.go:122] "RemoveContainer" containerID="d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea"
Nov 02 14:57:09.585020 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:57:09.584987    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea\": container with ID starting with d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea not found: ID does not exist" containerID="d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea"
Nov 02 14:57:09.585136 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.585037    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea"} err="failed to get container status \"d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea\": rpc error: code = NotFound desc = could not find container \"d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea\": container with ID starting with d5ee363cd0409648f36d01ed99821dfbab2a942a5de4016ea91ef6e39c410bea not found: ID does not exist"
Nov 02 14:57:09.624314 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.624271    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"]
Nov 02 14:57:09.633637 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:09.633590    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/sub-test-model-fcefcc-kserve-6b68557b85-gcf2c"]
Nov 02 14:57:10.171163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:10.171124    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb75de2-3838-4b3e-a5ab-85e2426628e7" path="/var/lib/kubelet/pods/0eb75de2-3838-4b3e-a5ab-85e2426628e7/volumes"
Nov 02 14:57:13.896927 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:57:13.896880    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:57:15.319726 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.319686    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"]
Nov 02 14:57:15.320722 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.319994    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls" podUID="262e5c36-6101-405c-899e-78712757602c" containerName="maas-api" containerID="cri-o://f3f55d629e28469eee57fb404d8ba7bbc0dc6c419cb6e1ed1472bff7fc074849" gracePeriod=30
Nov 02 14:57:15.624052 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.624012    6284 generic.go:356] "Generic (PLEG): container finished" podID="262e5c36-6101-405c-899e-78712757602c" containerID="f3f55d629e28469eee57fb404d8ba7bbc0dc6c419cb6e1ed1472bff7fc074849" exitCode=0
Nov 02 14:57:15.624258 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.624203    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls" event={"ID":"262e5c36-6101-405c-899e-78712757602c","Type":"ContainerDied","Data":"f3f55d629e28469eee57fb404d8ba7bbc0dc6c419cb6e1ed1472bff7fc074849"}
Nov 02 14:57:15.647349 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.647320    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:57:15.791673 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.791622    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262e5c36-6101-405c-899e-78712757602c-kube-api-access-t2q7x\" (UniqueName: \"kubernetes.io/projected/262e5c36-6101-405c-899e-78712757602c-kube-api-access-t2q7x\") pod \"262e5c36-6101-405c-899e-78712757602c\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") "
Nov 02 14:57:15.791986 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.791726    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-maas-api-tls\") pod \"262e5c36-6101-405c-899e-78712757602c\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") "
Nov 02 14:57:15.791986 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.791894    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262e5c36-6101-405c-899e-78712757602c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262e5c36-6101-405c-899e-78712757602c-openshift-service-ca\") pod \"262e5c36-6101-405c-899e-78712757602c\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") "
Nov 02 14:57:15.791986 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.791930    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-metrics-certs\") pod \"262e5c36-6101-405c-899e-78712757602c\" (UID: \"262e5c36-6101-405c-899e-78712757602c\") "
Nov 02 14:57:15.792279 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.792247    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262e5c36-6101-405c-899e-78712757602c-openshift-service-ca" pod "262e5c36-6101-405c-899e-78712757602c" (UID: "262e5c36-6101-405c-899e-78712757602c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:15.792396 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.792374    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/262e5c36-6101-405c-899e-78712757602c-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:15.794139 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.794104    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-maas-api-tls" pod "262e5c36-6101-405c-899e-78712757602c" (UID: "262e5c36-6101-405c-899e-78712757602c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:15.794260 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.794137    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262e5c36-6101-405c-899e-78712757602c-kube-api-access-t2q7x" pod "262e5c36-6101-405c-899e-78712757602c" (UID: "262e5c36-6101-405c-899e-78712757602c"). InnerVolumeSpecName "kube-api-access-t2q7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:15.794260 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.794245    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-metrics-certs" pod "262e5c36-6101-405c-899e-78712757602c" (UID: "262e5c36-6101-405c-899e-78712757602c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:15.893989 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.893838    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:15.893989 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.893882    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2q7x\" (UniqueName: \"kubernetes.io/projected/262e5c36-6101-405c-899e-78712757602c-kube-api-access-t2q7x\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:15.893989 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:15.893896    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/262e5c36-6101-405c-899e-78712757602c-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:16.631864 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:16.631828    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"
Nov 02 14:57:16.632372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:16.631866    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls" event={"ID":"262e5c36-6101-405c-899e-78712757602c","Type":"ContainerDied","Data":"9e9a398749134ba785c28ae735b9f021accaac6b5fbdcbcbaef43965c10d5f83"}
Nov 02 14:57:16.632372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:16.631946    6284 scope.go:122] "RemoveContainer" containerID="f3f55d629e28469eee57fb404d8ba7bbc0dc6c419cb6e1ed1472bff7fc074849"
Nov 02 14:57:16.676612 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:16.676568    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"]
Nov 02 14:57:16.680716 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:16.680680    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-ee3eca-6989bc757-5t9ls"]
Nov 02 14:57:18.169320 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:18.169288    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262e5c36-6101-405c-899e-78712757602c" path="/var/lib/kubelet/pods/262e5c36-6101-405c-899e-78712757602c/volumes"
Nov 02 14:57:26.348427 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.348389    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"]
Nov 02 14:57:26.349252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.349216    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw" podUID="274d4228-83fc-4efb-962b-63f1dbf48ac8" containerName="maas-api" containerID="cri-o://6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1" gracePeriod=30
Nov 02 14:57:26.659843 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.654788    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:57:26.700754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.700717    6284 generic.go:356] "Generic (PLEG): container finished" podID="274d4228-83fc-4efb-962b-63f1dbf48ac8" containerID="6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1" exitCode=0
Nov 02 14:57:26.700945 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.700785    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"
Nov 02 14:57:26.700945 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.700792    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw" event={"ID":"274d4228-83fc-4efb-962b-63f1dbf48ac8","Type":"ContainerDied","Data":"6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1"}
Nov 02 14:57:26.700945 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.700853    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw" event={"ID":"274d4228-83fc-4efb-962b-63f1dbf48ac8","Type":"ContainerDied","Data":"34bda03f9f34780a1a7350f346747166f4979f4df7a90ad65ec1519340eb0226"}
Nov 02 14:57:26.700945 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.700876    6284 scope.go:122] "RemoveContainer" containerID="6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1"
Nov 02 14:57:26.710672 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.710635    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274d4228-83fc-4efb-962b-63f1dbf48ac8-kube-api-access-vgsq2\" (UniqueName: \"kubernetes.io/projected/274d4228-83fc-4efb-962b-63f1dbf48ac8-kube-api-access-vgsq2\") pod \"274d4228-83fc-4efb-962b-63f1dbf48ac8\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") "
Nov 02 14:57:26.710868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.710708    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-maas-api-tls\") pod \"274d4228-83fc-4efb-962b-63f1dbf48ac8\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") "
Nov 02 14:57:26.710868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.710773    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-metrics-certs\") pod \"274d4228-83fc-4efb-962b-63f1dbf48ac8\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") "
Nov 02 14:57:26.710868 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.710811    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274d4228-83fc-4efb-962b-63f1dbf48ac8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/274d4228-83fc-4efb-962b-63f1dbf48ac8-openshift-service-ca\") pod \"274d4228-83fc-4efb-962b-63f1dbf48ac8\" (UID: \"274d4228-83fc-4efb-962b-63f1dbf48ac8\") "
Nov 02 14:57:26.711482 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.711446    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274d4228-83fc-4efb-962b-63f1dbf48ac8-openshift-service-ca" pod "274d4228-83fc-4efb-962b-63f1dbf48ac8" (UID: "274d4228-83fc-4efb-962b-63f1dbf48ac8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:26.714017 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.713417    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-metrics-certs" pod "274d4228-83fc-4efb-962b-63f1dbf48ac8" (UID: "274d4228-83fc-4efb-962b-63f1dbf48ac8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:26.714017 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.713525    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-maas-api-tls" pod "274d4228-83fc-4efb-962b-63f1dbf48ac8" (UID: "274d4228-83fc-4efb-962b-63f1dbf48ac8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:26.714414 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.714383    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274d4228-83fc-4efb-962b-63f1dbf48ac8-kube-api-access-vgsq2" pod "274d4228-83fc-4efb-962b-63f1dbf48ac8" (UID: "274d4228-83fc-4efb-962b-63f1dbf48ac8"). InnerVolumeSpecName "kube-api-access-vgsq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:26.715113 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.715091    6284 scope.go:122] "RemoveContainer" containerID="6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1"
Nov 02 14:57:26.715554 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:57:26.715532    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1\": container with ID starting with 6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1 not found: ID does not exist" containerID="6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1"
Nov 02 14:57:26.715647 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.715565    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1"} err="failed to get container status \"6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1\": rpc error: code = NotFound desc = could not find container \"6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1\": container with ID starting with 6636efe6eb3482fddf6191739876843c398d1f9f8e5f5148bf6f2b66b10f8ec1 not found: ID does not exist"
Nov 02 14:57:26.812464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.812424    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgsq2\" (UniqueName: \"kubernetes.io/projected/274d4228-83fc-4efb-962b-63f1dbf48ac8-kube-api-access-vgsq2\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:26.812464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.812460    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:26.812464 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.812470    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/274d4228-83fc-4efb-962b-63f1dbf48ac8-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:26.812721 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:26.812480    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/274d4228-83fc-4efb-962b-63f1dbf48ac8-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:27.051128 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:27.051086    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"]
Nov 02 14:57:27.060602 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:27.060561    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a02c35-6b6fdcfdbf-vl2hw"]
Nov 02 14:57:28.170789 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:28.170753    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274d4228-83fc-4efb-962b-63f1dbf48ac8" path="/var/lib/kubelet/pods/274d4228-83fc-4efb-962b-63f1dbf48ac8/volumes"
Nov 02 14:57:32.030260 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.030220    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"]
Nov 02 14:57:32.030740 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.030523    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv" podUID="3779b6eb-7714-44e3-a3ae-75e381312aed" containerName="maas-api" containerID="cri-o://c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407" gracePeriod=30
Nov 02 14:57:32.315899 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.315868    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:57:32.377432 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.377381    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-maas-api-tls\") pod \"3779b6eb-7714-44e3-a3ae-75e381312aed\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") "
Nov 02 14:57:32.377646 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.377454    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3779b6eb-7714-44e3-a3ae-75e381312aed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3779b6eb-7714-44e3-a3ae-75e381312aed-openshift-service-ca\") pod \"3779b6eb-7714-44e3-a3ae-75e381312aed\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") "
Nov 02 14:57:32.377646 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.377506    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3779b6eb-7714-44e3-a3ae-75e381312aed-kube-api-access-drbjk\" (UniqueName: \"kubernetes.io/projected/3779b6eb-7714-44e3-a3ae-75e381312aed-kube-api-access-drbjk\") pod \"3779b6eb-7714-44e3-a3ae-75e381312aed\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") "
Nov 02 14:57:32.377646 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.377592    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-metrics-certs\") pod \"3779b6eb-7714-44e3-a3ae-75e381312aed\" (UID: \"3779b6eb-7714-44e3-a3ae-75e381312aed\") "
Nov 02 14:57:32.377966 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.377905    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3779b6eb-7714-44e3-a3ae-75e381312aed-openshift-service-ca" pod "3779b6eb-7714-44e3-a3ae-75e381312aed" (UID: "3779b6eb-7714-44e3-a3ae-75e381312aed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:32.378232 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.378211    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3779b6eb-7714-44e3-a3ae-75e381312aed-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:32.380427 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.380388    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-maas-api-tls" pod "3779b6eb-7714-44e3-a3ae-75e381312aed" (UID: "3779b6eb-7714-44e3-a3ae-75e381312aed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:32.380584 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.380442    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-metrics-certs" pod "3779b6eb-7714-44e3-a3ae-75e381312aed" (UID: "3779b6eb-7714-44e3-a3ae-75e381312aed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:32.380700 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.380676    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3779b6eb-7714-44e3-a3ae-75e381312aed-kube-api-access-drbjk" pod "3779b6eb-7714-44e3-a3ae-75e381312aed" (UID: "3779b6eb-7714-44e3-a3ae-75e381312aed"). InnerVolumeSpecName "kube-api-access-drbjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:32.479189 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.479146    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:32.479189 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.479184    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3779b6eb-7714-44e3-a3ae-75e381312aed-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:32.479189 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.479194    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbjk\" (UniqueName: \"kubernetes.io/projected/3779b6eb-7714-44e3-a3ae-75e381312aed-kube-api-access-drbjk\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:32.743069 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.743034    6284 generic.go:356] "Generic (PLEG): container finished" podID="3779b6eb-7714-44e3-a3ae-75e381312aed" containerID="c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407" exitCode=0
Nov 02 14:57:32.743373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.743110    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"
Nov 02 14:57:32.743373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.743115    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv" event={"ID":"3779b6eb-7714-44e3-a3ae-75e381312aed","Type":"ContainerDied","Data":"c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407"}
Nov 02 14:57:32.743373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.743213    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv" event={"ID":"3779b6eb-7714-44e3-a3ae-75e381312aed","Type":"ContainerDied","Data":"38dd79117055950cc9226a0fc442ed33c818ae7713d177211dca380e249eb300"}
Nov 02 14:57:32.743373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.743246    6284 scope.go:122] "RemoveContainer" containerID="c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407"
Nov 02 14:57:32.755182 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.755155    6284 scope.go:122] "RemoveContainer" containerID="c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407"
Nov 02 14:57:32.755631 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:57:32.755602    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407\": container with ID starting with c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407 not found: ID does not exist" containerID="c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407"
Nov 02 14:57:32.755768 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.755702    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407"} err="failed to get container status \"c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407\": rpc error: code = NotFound desc = could not find container \"c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407\": container with ID starting with c26877725ca727a06c1fc542454c84ea6e3b1af2d8304e7c2b28d2cefeee9407 not found: ID does not exist"
Nov 02 14:57:32.789908 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.789867    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"]
Nov 02 14:57:32.796673 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:32.796638    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-21baa7-5b5db44f89-qp5wv"]
Nov 02 14:57:34.169664 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:34.169625    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3779b6eb-7714-44e3-a3ae-75e381312aed" path="/var/lib/kubelet/pods/3779b6eb-7714-44e3-a3ae-75e381312aed/volumes"
Nov 02 14:57:35.830584 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:35.830494    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"]
Nov 02 14:57:35.831163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:35.830764    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk" podUID="037046d9-f13b-4a27-a9d2-6b113f1a9fc6" containerName="maas-api" containerID="cri-o://1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3" gracePeriod=30
Nov 02 14:57:36.127270 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.127239    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:57:36.220504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.220465    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-maas-api-tls\") pod \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") "
Nov 02 14:57:36.220723 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.220526    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-kube-api-access-drnth\" (UniqueName: \"kubernetes.io/projected/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-kube-api-access-drnth\") pod \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") "
Nov 02 14:57:36.220723 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.220563    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-openshift-service-ca\") pod \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") "
Nov 02 14:57:36.220723 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.220620    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-metrics-certs\") pod \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\" (UID: \"037046d9-f13b-4a27-a9d2-6b113f1a9fc6\") "
Nov 02 14:57:36.221049 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.221019    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-openshift-service-ca" pod "037046d9-f13b-4a27-a9d2-6b113f1a9fc6" (UID: "037046d9-f13b-4a27-a9d2-6b113f1a9fc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:36.222873 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.222845    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-maas-api-tls" pod "037046d9-f13b-4a27-a9d2-6b113f1a9fc6" (UID: "037046d9-f13b-4a27-a9d2-6b113f1a9fc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:36.223041 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.222911    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-kube-api-access-drnth" pod "037046d9-f13b-4a27-a9d2-6b113f1a9fc6" (UID: "037046d9-f13b-4a27-a9d2-6b113f1a9fc6"). InnerVolumeSpecName "kube-api-access-drnth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:36.223041 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.222922    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-metrics-certs" pod "037046d9-f13b-4a27-a9d2-6b113f1a9fc6" (UID: "037046d9-f13b-4a27-a9d2-6b113f1a9fc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:36.324857 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.324809    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:36.324857 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.324854    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drnth\" (UniqueName: \"kubernetes.io/projected/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-kube-api-access-drnth\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:36.325134 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.324871    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:36.325134 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.324885    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/037046d9-f13b-4a27-a9d2-6b113f1a9fc6-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:36.768160 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.768126    6284 generic.go:356] "Generic (PLEG): container finished" podID="037046d9-f13b-4a27-a9d2-6b113f1a9fc6" containerID="1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3" exitCode=0
Nov 02 14:57:36.768326 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.768172    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"
Nov 02 14:57:36.768326 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.768210    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk" event={"ID":"037046d9-f13b-4a27-a9d2-6b113f1a9fc6","Type":"ContainerDied","Data":"1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3"}
Nov 02 14:57:36.768326 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.768268    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk" event={"ID":"037046d9-f13b-4a27-a9d2-6b113f1a9fc6","Type":"ContainerDied","Data":"df06e69df627967b5f2afcbc650af4a2fd6333a39a19d422b6a2920a2c7048b5"}
Nov 02 14:57:36.768326 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.768293    6284 scope.go:122] "RemoveContainer" containerID="1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3"
Nov 02 14:57:36.779558 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.779536    6284 scope.go:122] "RemoveContainer" containerID="1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3"
Nov 02 14:57:36.779919 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:57:36.779890    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3\": container with ID starting with 1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3 not found: ID does not exist" containerID="1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3"
Nov 02 14:57:36.780021 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.779947    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3"} err="failed to get container status \"1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3\": rpc error: code = NotFound desc = could not find container \"1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3\": container with ID starting with 1c8f58340263cee9bd9288b57d78c1368b480273bf6ab708b5eebb8bb13423f3 not found: ID does not exist"
Nov 02 14:57:36.820029 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.819988    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"]
Nov 02 14:57:36.829547 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:36.829513    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-2cddf2-55fc958487-6kgwk"]
Nov 02 14:57:38.170641 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:38.170606    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037046d9-f13b-4a27-a9d2-6b113f1a9fc6" path="/var/lib/kubelet/pods/037046d9-f13b-4a27-a9d2-6b113f1a9fc6/volumes"
Nov 02 14:57:43.896392 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:57:43.896350    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:57:53.538864 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.538772    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"]
Nov 02 14:57:53.539446 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.539279    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9" podUID="388bb99e-a56a-479d-964c-0e1dfaa5ec3a" containerName="maas-api" containerID="cri-o://e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095" gracePeriod=30
Nov 02 14:57:53.837388 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.837346    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:57:53.874017 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.873980    6284 generic.go:356] "Generic (PLEG): container finished" podID="388bb99e-a56a-479d-964c-0e1dfaa5ec3a" containerID="e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095" exitCode=0
Nov 02 14:57:53.874017 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.874020    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"
Nov 02 14:57:53.874252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.874064    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9" event={"ID":"388bb99e-a56a-479d-964c-0e1dfaa5ec3a","Type":"ContainerDied","Data":"e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095"}
Nov 02 14:57:53.874252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.874124    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9" event={"ID":"388bb99e-a56a-479d-964c-0e1dfaa5ec3a","Type":"ContainerDied","Data":"db21c6ada0806ad6d63269d50f878d6eba2a839fec11c42c74f3fed8b379da23"}
Nov 02 14:57:53.874252 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.874150    6284 scope.go:122] "RemoveContainer" containerID="e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095"
Nov 02 14:57:53.885900 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.885874    6284 scope.go:122] "RemoveContainer" containerID="e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095"
Nov 02 14:57:53.888029 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:57:53.886338    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095\": container with ID starting with e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095 not found: ID does not exist" containerID="e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095"
Nov 02 14:57:53.888029 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.886386    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095"} err="failed to get container status \"e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095\": rpc error: code = NotFound desc = could not find container \"e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095\": container with ID starting with e869223d2d614064c78b59d5555f3a40e0b5a9e03f8e9914ed87ef3291ec8095 not found: ID does not exist"
Nov 02 14:57:53.926115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.926069    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-openshift-service-ca\") pod \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") "
Nov 02 14:57:53.926115 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.926116    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-maas-api-tls\") pod \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") "
Nov 02 14:57:53.926401 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.926175    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-kube-api-access-mdkmd\" (UniqueName: \"kubernetes.io/projected/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-kube-api-access-mdkmd\") pod \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") "
Nov 02 14:57:53.926401 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.926198    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-metrics-certs\") pod \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\" (UID: \"388bb99e-a56a-479d-964c-0e1dfaa5ec3a\") "
Nov 02 14:57:53.926562 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.926533    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-openshift-service-ca" pod "388bb99e-a56a-479d-964c-0e1dfaa5ec3a" (UID: "388bb99e-a56a-479d-964c-0e1dfaa5ec3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:53.928499 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.928461    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-metrics-certs" pod "388bb99e-a56a-479d-964c-0e1dfaa5ec3a" (UID: "388bb99e-a56a-479d-964c-0e1dfaa5ec3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:53.928637 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.928507    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-kube-api-access-mdkmd" pod "388bb99e-a56a-479d-964c-0e1dfaa5ec3a" (UID: "388bb99e-a56a-479d-964c-0e1dfaa5ec3a"). InnerVolumeSpecName "kube-api-access-mdkmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:53.928637 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:53.928546    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-maas-api-tls" pod "388bb99e-a56a-479d-964c-0e1dfaa5ec3a" (UID: "388bb99e-a56a-479d-964c-0e1dfaa5ec3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:54.027909 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:54.027626    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:54.027909 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:54.027668    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:54.027909 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:54.027681    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdkmd\" (UniqueName: \"kubernetes.io/projected/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-kube-api-access-mdkmd\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:54.027909 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:54.027694    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/388bb99e-a56a-479d-964c-0e1dfaa5ec3a-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:57:54.211786 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:54.211744    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"]
Nov 02 14:57:54.217122 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:54.217088    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-4c2811-6ff9d9b4fb-486r9"]
Nov 02 14:57:56.169281 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:57:56.169231    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388bb99e-a56a-479d-964c-0e1dfaa5ec3a" path="/var/lib/kubelet/pods/388bb99e-a56a-479d-964c-0e1dfaa5ec3a/volumes"
Nov 02 14:58:13.896420 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:58:13.896374    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:58:14.361957 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.361891    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"]
Nov 02 14:58:14.362800 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.362738    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94" podUID="025e41b6-cbe6-4e32-9b90-57d466d13c22" containerName="maas-api" containerID="cri-o://b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f" gracePeriod=30
Nov 02 14:58:14.660227 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.660196    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:58:14.763385 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.763344    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-maas-api-tls\") pod \"025e41b6-cbe6-4e32-9b90-57d466d13c22\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") "
Nov 02 14:58:14.763620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.763446    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025e41b6-cbe6-4e32-9b90-57d466d13c22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/025e41b6-cbe6-4e32-9b90-57d466d13c22-openshift-service-ca\") pod \"025e41b6-cbe6-4e32-9b90-57d466d13c22\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") "
Nov 02 14:58:14.763620 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.763507    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-metrics-certs\") pod \"025e41b6-cbe6-4e32-9b90-57d466d13c22\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") "
Nov 02 14:58:14.763767 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.763640    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025e41b6-cbe6-4e32-9b90-57d466d13c22-kube-api-access-db7g4\" (UniqueName: \"kubernetes.io/projected/025e41b6-cbe6-4e32-9b90-57d466d13c22-kube-api-access-db7g4\") pod \"025e41b6-cbe6-4e32-9b90-57d466d13c22\" (UID: \"025e41b6-cbe6-4e32-9b90-57d466d13c22\") "
Nov 02 14:58:14.763905 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.763874    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025e41b6-cbe6-4e32-9b90-57d466d13c22-openshift-service-ca" pod "025e41b6-cbe6-4e32-9b90-57d466d13c22" (UID: "025e41b6-cbe6-4e32-9b90-57d466d13c22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:14.764069 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.764047    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/025e41b6-cbe6-4e32-9b90-57d466d13c22-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:14.765897 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.765860    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025e41b6-cbe6-4e32-9b90-57d466d13c22-kube-api-access-db7g4" pod "025e41b6-cbe6-4e32-9b90-57d466d13c22" (UID: "025e41b6-cbe6-4e32-9b90-57d466d13c22"). InnerVolumeSpecName "kube-api-access-db7g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:14.766203 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.766173    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-metrics-certs" pod "025e41b6-cbe6-4e32-9b90-57d466d13c22" (UID: "025e41b6-cbe6-4e32-9b90-57d466d13c22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:14.766318 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.766185    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-maas-api-tls" pod "025e41b6-cbe6-4e32-9b90-57d466d13c22" (UID: "025e41b6-cbe6-4e32-9b90-57d466d13c22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:14.865433 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.865396    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db7g4\" (UniqueName: \"kubernetes.io/projected/025e41b6-cbe6-4e32-9b90-57d466d13c22-kube-api-access-db7g4\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:14.865433 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.865432    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:14.865433 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:14.865443    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/025e41b6-cbe6-4e32-9b90-57d466d13c22-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:15.007657 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.007617    6284 generic.go:356] "Generic (PLEG): container finished" podID="025e41b6-cbe6-4e32-9b90-57d466d13c22" containerID="b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f" exitCode=0
Nov 02 14:58:15.008116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.007694    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"
Nov 02 14:58:15.008116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.007700    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94" event={"ID":"025e41b6-cbe6-4e32-9b90-57d466d13c22","Type":"ContainerDied","Data":"b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f"}
Nov 02 14:58:15.008116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.007785    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94" event={"ID":"025e41b6-cbe6-4e32-9b90-57d466d13c22","Type":"ContainerDied","Data":"eac49772c0c7834ce94ee922ba6f63a4b62163062dd1f3aa5faac2924f12f8f3"}
Nov 02 14:58:15.008116 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.007807    6284 scope.go:122] "RemoveContainer" containerID="b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f"
Nov 02 14:58:15.018674 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.018653    6284 scope.go:122] "RemoveContainer" containerID="76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6"
Nov 02 14:58:15.028747 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.028722    6284 scope.go:122] "RemoveContainer" containerID="b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f"
Nov 02 14:58:15.029162 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:58:15.029139    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f\": container with ID starting with b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f not found: ID does not exist" containerID="b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f"
Nov 02 14:58:15.029253 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.029178    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f"} err="failed to get container status \"b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f\": rpc error: code = NotFound desc = could not find container \"b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f\": container with ID starting with b33454db4481715020a585702dfea42354c3b41a93ece38804ae8fc4b4a1ba7f not found: ID does not exist"
Nov 02 14:58:15.029253 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.029201    6284 scope.go:122] "RemoveContainer" containerID="76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6"
Nov 02 14:58:15.029480 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:58:15.029461    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6\": container with ID starting with 76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6 not found: ID does not exist" containerID="76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6"
Nov 02 14:58:15.029538 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.029490    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6"} err="failed to get container status \"76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6\": rpc error: code = NotFound desc = could not find container \"76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6\": container with ID starting with 76faed08d66e9fd73b2008a96c9e2846c3de8551916e800a0b77839468c4fdb6 not found: ID does not exist"
Nov 02 14:58:15.053598 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.053553    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"]
Nov 02 14:58:15.063192 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:15.063155    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fcefcc-8569998f57-xht94"]
Nov 02 14:58:16.169552 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:16.169516    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025e41b6-cbe6-4e32-9b90-57d466d13c22" path="/var/lib/kubelet/pods/025e41b6-cbe6-4e32-9b90-57d466d13c22/volumes"
Nov 02 14:58:18.489824 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:18.489453    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"]
Nov 02 14:58:18.489824 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:18.489738    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerName="istio-proxy" containerID="cri-o://8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9" gracePeriod=30
Nov 02 14:58:19.826732 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:19.826685    6284 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:58:19.827202 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:19.826761    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:58:20.377373 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.377329    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"]
Nov 02 14:58:20.377619 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.377590    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs" podUID="5c109341-915a-457e-a0b0-e9351acad498" containerName="maas-api" containerID="cri-o://5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be" gracePeriod=30
Nov 02 14:58:20.665449 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.665409    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 14:58:20.839401 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.839355    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c109341-915a-457e-a0b0-e9351acad498-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c109341-915a-457e-a0b0-e9351acad498-openshift-service-ca\") pod \"5c109341-915a-457e-a0b0-e9351acad498\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") "
Nov 02 14:58:20.839907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.839434    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-metrics-certs\") pod \"5c109341-915a-457e-a0b0-e9351acad498\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") "
Nov 02 14:58:20.839907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.839538    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-maas-api-tls\") pod \"5c109341-915a-457e-a0b0-e9351acad498\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") "
Nov 02 14:58:20.839907 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.839654    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c109341-915a-457e-a0b0-e9351acad498-kube-api-access-9j97r\" (UniqueName: \"kubernetes.io/projected/5c109341-915a-457e-a0b0-e9351acad498-kube-api-access-9j97r\") pod \"5c109341-915a-457e-a0b0-e9351acad498\" (UID: \"5c109341-915a-457e-a0b0-e9351acad498\") "
Nov 02 14:58:20.840112 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.839900    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c109341-915a-457e-a0b0-e9351acad498-openshift-service-ca" pod "5c109341-915a-457e-a0b0-e9351acad498" (UID: "5c109341-915a-457e-a0b0-e9351acad498"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:20.842672 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.842625    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-maas-api-tls" pod "5c109341-915a-457e-a0b0-e9351acad498" (UID: "5c109341-915a-457e-a0b0-e9351acad498"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:20.845480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.845442    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c109341-915a-457e-a0b0-e9351acad498-kube-api-access-9j97r" pod "5c109341-915a-457e-a0b0-e9351acad498" (UID: "5c109341-915a-457e-a0b0-e9351acad498"). InnerVolumeSpecName "kube-api-access-9j97r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:20.845480 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.845468    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-metrics-certs" pod "5c109341-915a-457e-a0b0-e9351acad498" (UID: "5c109341-915a-457e-a0b0-e9351acad498"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:20.941227 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.941129    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j97r\" (UniqueName: \"kubernetes.io/projected/5c109341-915a-457e-a0b0-e9351acad498-kube-api-access-9j97r\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:20.941227 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.941171    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c109341-915a-457e-a0b0-e9351acad498-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:20.941227 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.941182    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:20.941227 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:20.941191    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c109341-915a-457e-a0b0-e9351acad498-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:21.046661 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:21.046625    6284 generic.go:356] "Generic (PLEG): container finished" podID="5c109341-915a-457e-a0b0-e9351acad498" containerID="5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be" exitCode=0
Nov 02 14:58:21.046854 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:21.046667    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"
Nov 02 14:58:21.046854 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:21.046699    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs" event={"ID":"5c109341-915a-457e-a0b0-e9351acad498","Type":"ContainerDied","Data":"5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be"}
Nov 02 14:58:21.046854 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:21.046775    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs" event={"ID":"5c109341-915a-457e-a0b0-e9351acad498","Type":"ContainerDied","Data":"037d5b0a4c24598c2e6a6109682421b6ed0404a3b2bb3164d117c61a2b6974b3"}
Nov 02 14:58:21.046854 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:21.046800    6284 scope.go:122] "RemoveContainer" containerID="5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be"
Nov 02 14:58:21.057081 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:21.057058    6284 scope.go:122] "RemoveContainer" containerID="5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be"
Nov 02 14:58:21.057469 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:58:21.057442    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be\": container with ID starting with 5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be not found: ID does not exist" containerID="5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be"
Nov 02 14:58:21.057533 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:21.057481    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be"} err="failed to get container status \"5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be\": rpc error: code = NotFound desc = could not find container \"5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be\": container with ID starting with 5594b385c442e24befcb94dec3b66a0b08a7d58488c749afb2bb50717f8599be not found: ID does not exist"
Nov 02 14:58:21.096538 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:21.096498    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"]
Nov 02 14:58:21.107267 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:21.107232    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-92f249-6fb74fd99b-7ssfs"]
Nov 02 14:58:22.169095 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:22.169052    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c109341-915a-457e-a0b0-e9351acad498" path="/var/lib/kubelet/pods/5c109341-915a-457e-a0b0-e9351acad498/volumes"
Nov 02 14:58:23.763369 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.763295    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:58:23.873515 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.873475    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-kube-api-access-q7p28\" (UniqueName: \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-kube-api-access-q7p28\") pod \"ca340223-2a40-4e94-a02f-8031a515cc1d\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") "
Nov 02 14:58:23.873714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.873526    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-data\") pod \"ca340223-2a40-4e94-a02f-8031a515cc1d\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") "
Nov 02 14:58:23.873714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.873577    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-token\" (UniqueName: \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-token\") pod \"ca340223-2a40-4e94-a02f-8031a515cc1d\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") "
Nov 02 14:58:23.873714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.873607    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-podinfo\") pod \"ca340223-2a40-4e94-a02f-8031a515cc1d\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") "
Nov 02 14:58:23.873714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.873673    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-envoy\") pod \"ca340223-2a40-4e94-a02f-8031a515cc1d\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") "
Nov 02 14:58:23.873714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.873690    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca340223-2a40-4e94-a02f-8031a515cc1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca340223-2a40-4e94-a02f-8031a515cc1d-istiod-ca-cert\") pod \"ca340223-2a40-4e94-a02f-8031a515cc1d\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") "
Nov 02 14:58:23.873714 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.873712    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-socket\") pod \"ca340223-2a40-4e94-a02f-8031a515cc1d\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") "
Nov 02 14:58:23.874251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.873764    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-credential-socket\") pod \"ca340223-2a40-4e94-a02f-8031a515cc1d\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") "
Nov 02 14:58:23.874251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.873792    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-certs\") pod \"ca340223-2a40-4e94-a02f-8031a515cc1d\" (UID: \"ca340223-2a40-4e94-a02f-8031a515cc1d\") "
Nov 02 14:58:23.874251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.873887    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-data" pod "ca340223-2a40-4e94-a02f-8031a515cc1d" (UID: "ca340223-2a40-4e94-a02f-8031a515cc1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:23.874251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.874205    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-socket" pod "ca340223-2a40-4e94-a02f-8031a515cc1d" (UID: "ca340223-2a40-4e94-a02f-8031a515cc1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:23.874251 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.874224    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-credential-socket" pod "ca340223-2a40-4e94-a02f-8031a515cc1d" (UID: "ca340223-2a40-4e94-a02f-8031a515cc1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:23.874628 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.874332    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.874628 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.874404    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.874628 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.874438    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-certs" pod "ca340223-2a40-4e94-a02f-8031a515cc1d" (UID: "ca340223-2a40-4e94-a02f-8031a515cc1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:23.874628 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.874482    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca340223-2a40-4e94-a02f-8031a515cc1d-istiod-ca-cert" pod "ca340223-2a40-4e94-a02f-8031a515cc1d" (UID: "ca340223-2a40-4e94-a02f-8031a515cc1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:23.876101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.876066    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-podinfo" pod "ca340223-2a40-4e94-a02f-8031a515cc1d" (UID: "ca340223-2a40-4e94-a02f-8031a515cc1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:58:23.876230 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.876140    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-token" pod "ca340223-2a40-4e94-a02f-8031a515cc1d" (UID: "ca340223-2a40-4e94-a02f-8031a515cc1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:23.876230 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.876179    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-envoy" pod "ca340223-2a40-4e94-a02f-8031a515cc1d" (UID: "ca340223-2a40-4e94-a02f-8031a515cc1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:23.876230 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.876219    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-kube-api-access-q7p28" pod "ca340223-2a40-4e94-a02f-8031a515cc1d" (UID: "ca340223-2a40-4e94-a02f-8031a515cc1d"). InnerVolumeSpecName "kube-api-access-q7p28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:23.975643 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.975602    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.975643 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.975643    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.975643 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.975656    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.975915 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.975669    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca340223-2a40-4e94-a02f-8031a515cc1d-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.975915 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.975681    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.975915 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.975694    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca340223-2a40-4e94-a02f-8031a515cc1d-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.975915 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:23.975706    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7p28\" (UniqueName: \"kubernetes.io/projected/ca340223-2a40-4e94-a02f-8031a515cc1d-kube-api-access-q7p28\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:24.065691 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.065590    6284 generic.go:356] "Generic (PLEG): container finished" podID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerID="8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9" exitCode=0
Nov 02 14:58:24.065691 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.065653    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"
Nov 02 14:58:24.065929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.065658    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" event={"ID":"ca340223-2a40-4e94-a02f-8031a515cc1d","Type":"ContainerDied","Data":"8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9"}
Nov 02 14:58:24.065929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.065763    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln" event={"ID":"ca340223-2a40-4e94-a02f-8031a515cc1d","Type":"ContainerDied","Data":"c4c61ab87a90984ed70c73abcd43ce4dad1fc621b55ca12c85b261ca04b67376"}
Nov 02 14:58:24.065929 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.065788    6284 scope.go:122] "RemoveContainer" containerID="8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9"
Nov 02 14:58:24.076875 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.076848    6284 scope.go:122] "RemoveContainer" containerID="8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9"
Nov 02 14:58:24.077269 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:58:24.077240    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9\": container with ID starting with 8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9 not found: ID does not exist" containerID="8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9"
Nov 02 14:58:24.077341 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.077282    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9"} err="failed to get container status \"8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9\": rpc error: code = NotFound desc = could not find container \"8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9\": container with ID starting with 8e0fa89a643977131e020203bb3634b94d2733415d725622c12112451c6a7cf9 not found: ID does not exist"
Nov 02 14:58:24.126616 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.126571    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"]
Nov 02 14:58:24.147308 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.147270    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"]
Nov 02 14:58:24.153837 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.153796    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-czrln"]
Nov 02 14:58:24.169930 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.169895    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" path="/var/lib/kubelet/pods/ca340223-2a40-4e94-a02f-8031a515cc1d/volumes"
Nov 02 14:58:24.303754 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.303716    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"]
Nov 02 14:58:24.304413 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304387    6284 cpu_manager.go:414] "containerMap: removing container" podUID="261ca16d-584d-4b15-b34c-0a54eb966013" containerName="main"
Nov 02 14:58:24.304413 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304412    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261ca16d-584d-4b15-b34c-0a54eb966013" containerName="main"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304452    6284 cpu_manager.go:414] "containerMap: removing container" podUID="0eb75de2-3838-4b3e-a5ab-85e2426628e7" containerName="main"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304461    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb75de2-3838-4b3e-a5ab-85e2426628e7" containerName="main"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304473    6284 cpu_manager.go:414] "containerMap: removing container" podUID="91688d53-63bb-4a17-91c8-8e9d5f722eb4" containerName="payload-pre-processing"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304482    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91688d53-63bb-4a17-91c8-8e9d5f722eb4" containerName="payload-pre-processing"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304499    6284 cpu_manager.go:414] "containerMap: removing container" podUID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" containerName="istio-proxy"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304508    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" containerName="istio-proxy"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304522    6284 cpu_manager.go:414] "containerMap: removing container" podUID="c29c8cf7-2a01-4edc-98be-c7bd46bff630" containerName="istio-proxy"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304528    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29c8cf7-2a01-4edc-98be-c7bd46bff630" containerName="istio-proxy"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304553    6284 cpu_manager.go:414] "containerMap: removing container" podUID="fc70a0fe-2fe3-41be-b674-f96892b5ae58" containerName="maas-api"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304560    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc70a0fe-2fe3-41be-b674-f96892b5ae58" containerName="maas-api"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304572    6284 cpu_manager.go:414] "containerMap: removing container" podUID="025e41b6-cbe6-4e32-9b90-57d466d13c22" containerName="maas-api"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304579    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025e41b6-cbe6-4e32-9b90-57d466d13c22" containerName="maas-api"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304592    6284 cpu_manager.go:414] "containerMap: removing container" podUID="3779b6eb-7714-44e3-a3ae-75e381312aed" containerName="maas-api"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304598    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3779b6eb-7714-44e3-a3ae-75e381312aed" containerName="maas-api"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304609    6284 cpu_manager.go:414] "containerMap: removing container" podUID="262e5c36-6101-405c-899e-78712757602c" containerName="maas-api"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304615    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262e5c36-6101-405c-899e-78712757602c" containerName="maas-api"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304626    6284 cpu_manager.go:414] "containerMap: removing container" podUID="1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" containerName="main"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304632    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" containerName="main"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304643    6284 cpu_manager.go:414] "containerMap: removing container" podUID="025e41b6-cbe6-4e32-9b90-57d466d13c22" containerName="maas-api"
Nov 02 14:58:24.304639 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304649    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025e41b6-cbe6-4e32-9b90-57d466d13c22" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304660    6284 cpu_manager.go:414] "containerMap: removing container" podUID="baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" containerName="main"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304667    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" containerName="main"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304677    6284 cpu_manager.go:414] "containerMap: removing container" podUID="6ef217bd-b6ff-4c32-9591-8584e524bcd5" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304684    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef217bd-b6ff-4c32-9591-8584e524bcd5" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304736    6284 cpu_manager.go:414] "containerMap: removing container" podUID="adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" containerName="main"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304744    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" containerName="main"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304757    6284 cpu_manager.go:414] "containerMap: removing container" podUID="5c109341-915a-457e-a0b0-e9351acad498" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304764    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c109341-915a-457e-a0b0-e9351acad498" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304773    6284 cpu_manager.go:414] "containerMap: removing container" podUID="037046d9-f13b-4a27-a9d2-6b113f1a9fc6" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304780    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037046d9-f13b-4a27-a9d2-6b113f1a9fc6" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304791    6284 cpu_manager.go:414] "containerMap: removing container" podUID="92850818-3df8-445f-acda-7c17bf8b0577" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304797    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92850818-3df8-445f-acda-7c17bf8b0577" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304816    6284 cpu_manager.go:414] "containerMap: removing container" podUID="274d4228-83fc-4efb-962b-63f1dbf48ac8" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304823    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274d4228-83fc-4efb-962b-63f1dbf48ac8" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304836    6284 cpu_manager.go:414] "containerMap: removing container" podUID="388bb99e-a56a-479d-964c-0e1dfaa5ec3a" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304843    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388bb99e-a56a-479d-964c-0e1dfaa5ec3a" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304858    6284 cpu_manager.go:414] "containerMap: removing container" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerName="istio-proxy"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304865    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerName="istio-proxy"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304880    6284 cpu_manager.go:414] "containerMap: removing container" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.304887    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305031    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="baeba8f7-5d2a-47cc-8138-0ae3e4c8fffd" containerName="main"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305049    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef217bd-b6ff-4c32-9591-8584e524bcd5" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305059    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbf61d4-f51d-4ef7-8eda-c2a88a0ad96e" containerName="main"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305069    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c109341-915a-457e-a0b0-e9351acad498" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305076    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="037046d9-f13b-4a27-a9d2-6b113f1a9fc6" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305084    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="92850818-3df8-445f-acda-7c17bf8b0577" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305094    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="274d4228-83fc-4efb-962b-63f1dbf48ac8" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305104    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="388bb99e-a56a-479d-964c-0e1dfaa5ec3a" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305112    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca340223-2a40-4e94-a02f-8031a515cc1d" containerName="istio-proxy"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305123    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0db0e10-3bf4-46f5-be72-2ec3687ffcfa" containerName="istio-proxy"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305134    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="261ca16d-584d-4b15-b34c-0a54eb966013" containerName="main"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305142    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb75de2-3838-4b3e-a5ab-85e2426628e7" containerName="main"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305155    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="91688d53-63bb-4a17-91c8-8e9d5f722eb4" containerName="payload-pre-processing"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305166    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8fc299a-3d6d-4da3-94de-a974f6e3afc1" containerName="istio-proxy"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305174    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29c8cf7-2a01-4edc-98be-c7bd46bff630" containerName="istio-proxy"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305186    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc70a0fe-2fe3-41be-b674-f96892b5ae58" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305195    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="025e41b6-cbe6-4e32-9b90-57d466d13c22" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305204    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="3779b6eb-7714-44e3-a3ae-75e381312aed" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305215    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="262e5c36-6101-405c-899e-78712757602c" containerName="maas-api"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305224    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af7bc01-2fb9-45c4-9b6e-1e9ee016bda0" containerName="main"
Nov 02 14:58:24.305506 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.305234    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="025e41b6-cbe6-4e32-9b90-57d466d13c22" containerName="maas-api"
Nov 02 14:58:24.310867 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.310838    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.340452 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.340402    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"]
Nov 02 14:58:24.402721 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.402688    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ad42e0-openshift-default-dockercfg-gmvz9\""
Nov 02 14:58:24.480693 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.480659    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.480693 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.480699    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.480973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.480731    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.480973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.480887    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.480973 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.480945    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.481083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.481019    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.481083 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.481076    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxl2l\" (UniqueName: \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-kube-api-access-hxl2l\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.481151 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.481096    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.481197 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.481178    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.582505 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.582402    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.582505 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.582444    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.582505 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.582467    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.582829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.582513    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.582829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.582539    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.582829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.582562    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.582829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.582584    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxl2l\" (UniqueName: \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-kube-api-access-hxl2l\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.582829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.582605    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.582829 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.582667    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.583152 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.582911    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.583152 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.583039    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.583152 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.583081    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.583317 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.583252    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.583465 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.583438    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.585121 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.585095    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.585332 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.585316    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.604800 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.604764    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.604991 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.604870    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxl2l\" (UniqueName: \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-kube-api-access-hxl2l\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.630022 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.629977    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:24.948563 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.948475    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"]
Nov 02 14:58:24.953091 ip-10-0-131-98 kubenswrapper[6284]: W1102 14:58:24.953049    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025e5b6b_01b4_4e30_b6ee_8f496082cde5.slice/crio-df9232b9f6334a9ec6cca20c3b4451105793e7cdbc8e9fc39eb138ec5f334ab8 WatchSource:0}: Error finding container df9232b9f6334a9ec6cca20c3b4451105793e7cdbc8e9fc39eb138ec5f334ab8: Status 404 returned error can't find the container with id df9232b9f6334a9ec6cca20c3b4451105793e7cdbc8e9fc39eb138ec5f334ab8
Nov 02 14:58:24.964881 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.964842    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:58:24.965028 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.964917    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:58:24.965028 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:24.964986    6284 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 02 14:58:25.075670 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:25.075629    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" event={"ID":"025e5b6b-01b4-4e30-b6ee-8f496082cde5","Type":"ContainerStarted","Data":"4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310"}
Nov 02 14:58:25.075870 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:25.075723    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" event={"ID":"025e5b6b-01b4-4e30-b6ee-8f496082cde5","Type":"ContainerStarted","Data":"df9232b9f6334a9ec6cca20c3b4451105793e7cdbc8e9fc39eb138ec5f334ab8"}
Nov 02 14:58:25.122589 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:25.121446    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" podStartSLOduration=1.121422247 podStartE2EDuration="1.121422247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:25.118669516 +0000 UTC m=+4437.078045135" watchObservedRunningTime="2026-11-02 14:58:25.121422247 +0000 UTC m=+4437.080797855"
Nov 02 14:58:25.630846 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:25.630791    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:26.632358 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:26.632324    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:58:26.632768 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:26.632394    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 14:58:27.630626 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:27.630582    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:58:27.630860 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:27.630648    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 14:58:28.081401 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.081303    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"]
Nov 02 14:58:28.081830 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.081572    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q" podUID="9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1" containerName="maas-api" containerID="cri-o://8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d" gracePeriod=30
Nov 02 14:58:28.430615 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.430564    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 14:58:28.535582 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.535538    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-maas-api-tls\") pod \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") "
Nov 02 14:58:28.535767 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.535609    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-openshift-service-ca\") pod \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") "
Nov 02 14:58:28.535767 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.535672    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-metrics-certs\") pod \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") "
Nov 02 14:58:28.535767 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.535737    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-kube-api-access-t6qdg\" (UniqueName: \"kubernetes.io/projected/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-kube-api-access-t6qdg\") pod \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\" (UID: \"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1\") "
Nov 02 14:58:28.536281 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.536258    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-openshift-service-ca" pod "9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1" (UID: "9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:28.538448 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.538413    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-metrics-certs" pod "9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1" (UID: "9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:28.538573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.538430    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-maas-api-tls" pod "9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1" (UID: "9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:28.538573 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.538482    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-kube-api-access-t6qdg" pod "9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1" (UID: "9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1"). InnerVolumeSpecName "kube-api-access-t6qdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:28.631285 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.631236    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:58:28.631523 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.631298    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 14:58:28.636902 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.636864    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:28.636902 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.636898    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:28.636902 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.636910    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:28.637265 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:28.636919    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6qdg\" (UniqueName: \"kubernetes.io/projected/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1-kube-api-access-t6qdg\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:29.111706 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.111652    6284 generic.go:356] "Generic (PLEG): container finished" podID="9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1" containerID="8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d" exitCode=0
Nov 02 14:58:29.112262 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.111754    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"
Nov 02 14:58:29.112262 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.111975    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q" event={"ID":"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1","Type":"ContainerDied","Data":"8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d"}
Nov 02 14:58:29.112262 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.112036    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q" event={"ID":"9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1","Type":"ContainerDied","Data":"fb812a8ee1b6c3434d44241e627ebbd3d70fbf24df61d21c53ad372aa1ed48b4"}
Nov 02 14:58:29.112262 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.112072    6284 scope.go:122] "RemoveContainer" containerID="8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d"
Nov 02 14:58:29.127432 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.127391    6284 scope.go:122] "RemoveContainer" containerID="8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d"
Nov 02 14:58:29.127857 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:58:29.127829    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d\": container with ID starting with 8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d not found: ID does not exist" containerID="8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d"
Nov 02 14:58:29.127985 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.127865    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d"} err="failed to get container status \"8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d\": rpc error: code = NotFound desc = could not find container \"8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d\": container with ID starting with 8d43ee7985efaee477763fc4ae9c74ac814c4a3b93e2e2ff1461d5ac1d6e316d not found: ID does not exist"
Nov 02 14:58:29.178585 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.178539    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"]
Nov 02 14:58:29.178910 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.178878    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ad42e0-5466cc7bdc-89x2q"]
Nov 02 14:58:29.631194 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.631152    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:58:29.631386 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:29.631212    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 14:58:30.178483 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:30.178440    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1" path="/var/lib/kubelet/pods/9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1/volumes"
Nov 02 14:58:30.631163 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:30.631122    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:58:30.631362 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:30.631189    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 14:58:31.631411 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:31.631368    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:58:31.631958 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:31.631439    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Nov 02 14:58:33.672152 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:33.672102    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:58:33.672738 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:33.672169    6284 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:58:34.123400 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:34.123350    6284 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:34.123683 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:34.123660    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:34.124523 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:34.124500    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:39.149010 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.148921    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main" containerID="cri-o://9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c" gracePeriod=45
Nov 02 14:58:39.448879 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.448848    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 14:58:39.598802 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.598760    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-tmp-dir\") pod \"b3c53689-f9e1-42f5-b613-27c0f318f778\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") "
Nov 02 14:58:39.599018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.598877    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c53689-f9e1-42f5-b613-27c0f318f778-kube-api-access-nxzld\" (UniqueName: \"kubernetes.io/projected/b3c53689-f9e1-42f5-b613-27c0f318f778-kube-api-access-nxzld\") pod \"b3c53689-f9e1-42f5-b613-27c0f318f778\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") "
Nov 02 14:58:39.599018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.598947    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-dshm\") pod \"b3c53689-f9e1-42f5-b613-27c0f318f778\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") "
Nov 02 14:58:39.599018 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.598997    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3c53689-f9e1-42f5-b613-27c0f318f778-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c53689-f9e1-42f5-b613-27c0f318f778-tls-certs\") pod \"b3c53689-f9e1-42f5-b613-27c0f318f778\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") "
Nov 02 14:58:39.599206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.599028    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-home\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-home\") pod \"b3c53689-f9e1-42f5-b613-27c0f318f778\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") "
Nov 02 14:58:39.599206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.599066    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-tmp-dir" pod "b3c53689-f9e1-42f5-b613-27c0f318f778" (UID: "b3c53689-f9e1-42f5-b613-27c0f318f778"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:39.599206 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.599077    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-model-cache\") pod \"b3c53689-f9e1-42f5-b613-27c0f318f778\" (UID: \"b3c53689-f9e1-42f5-b613-27c0f318f778\") "
Nov 02 14:58:39.599549 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.599492    6284 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-tmp-dir\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.599685 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.599493    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-model-cache" pod "b3c53689-f9e1-42f5-b613-27c0f318f778" (UID: "b3c53689-f9e1-42f5-b613-27c0f318f778"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:39.599685 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.599558    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-home" pod "b3c53689-f9e1-42f5-b613-27c0f318f778" (UID: "b3c53689-f9e1-42f5-b613-27c0f318f778"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:39.601852 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.601814    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3c53689-f9e1-42f5-b613-27c0f318f778-tls-certs" pod "b3c53689-f9e1-42f5-b613-27c0f318f778" (UID: "b3c53689-f9e1-42f5-b613-27c0f318f778"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:39.602101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.601869    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c53689-f9e1-42f5-b613-27c0f318f778-kube-api-access-nxzld" pod "b3c53689-f9e1-42f5-b613-27c0f318f778" (UID: "b3c53689-f9e1-42f5-b613-27c0f318f778"). InnerVolumeSpecName "kube-api-access-nxzld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:39.602101 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.601875    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-dshm" pod "b3c53689-f9e1-42f5-b613-27c0f318f778" (UID: "b3c53689-f9e1-42f5-b613-27c0f318f778"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:39.700193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.700107    6284 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-dshm\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.700193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.700138    6284 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3c53689-f9e1-42f5-b613-27c0f318f778-tls-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.700193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.700148    6284 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-home\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.700193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.700156    6284 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3c53689-f9e1-42f5-b613-27c0f318f778-model-cache\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.700193 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:39.700165    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxzld\" (UniqueName: \"kubernetes.io/projected/b3c53689-f9e1-42f5-b613-27c0f318f778-kube-api-access-nxzld\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:40.180855 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:40.180820    6284 generic.go:356] "Generic (PLEG): container finished" podID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerID="9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c" exitCode=0
Nov 02 14:58:40.181327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:40.180880    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"
Nov 02 14:58:40.181327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:40.180879    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" event={"ID":"b3c53689-f9e1-42f5-b613-27c0f318f778","Type":"ContainerDied","Data":"9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c"}
Nov 02 14:58:40.181327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:40.180960    6284 scope.go:122] "RemoveContainer" containerID="9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c"
Nov 02 14:58:40.181327 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:40.180964    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj" event={"ID":"b3c53689-f9e1-42f5-b613-27c0f318f778","Type":"ContainerDied","Data":"8d5335571c625f32be02002f6b8648ba08a60b7c4a1364163c07f3f31abdf8a0"}
Nov 02 14:58:40.191600 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:40.191574    6284 scope.go:122] "RemoveContainer" containerID="9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c"
Nov 02 14:58:40.191969 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:58:40.191925    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c\": container with ID starting with 9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c not found: ID does not exist" containerID="9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c"
Nov 02 14:58:40.192071 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:40.191985    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c"} err="failed to get container status \"9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c\": rpc error: code = NotFound desc = could not find container \"9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c\": container with ID starting with 9239eb3fcac8d4f0920a8bd353dd9a2de7ce84d312d5ab0b8947b51cceb8bb8c not found: ID does not exist"
Nov 02 14:58:40.225527 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:40.225486    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"]
Nov 02 14:58:40.235369 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:40.235329    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/distinct-w2-0969b1-kserve-7c99747844-2hfmj"]
Nov 02 14:58:42.193087 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:42.193041    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" path="/var/lib/kubelet/pods/b3c53689-f9e1-42f5-b613-27c0f318f778/volumes"
Nov 02 14:58:42.225513 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:42.225472    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"]
Nov 02 14:58:42.225752 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:42.225724    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy" containerID="cri-o://c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228" gracePeriod=30
Nov 02 14:58:43.895004 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:58:43.894959    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:58:44.928645 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:44.928602    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"]
Nov 02 14:58:44.929198 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:44.929010    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy" containerID="cri-o://4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310" gracePeriod=30
Nov 02 14:58:45.257908 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:45.257866    6284 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:58:45.258111 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:45.257962    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:58:47.502544 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.502460    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:58:47.592068 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592016    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istiod-ca-cert\") pod \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") "
Nov 02 14:58:47.592288 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592130    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-token\" (UniqueName: \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-token\") pod \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") "
Nov 02 14:58:47.592372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592293    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-envoy\") pod \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") "
Nov 02 14:58:47.592372 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592332    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-data\") pod \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") "
Nov 02 14:58:47.592495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592391    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-credential-socket\") pod \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") "
Nov 02 14:58:47.592495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592441    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-podinfo\") pod \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") "
Nov 02 14:58:47.592495 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592478    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-socket\") pod \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") "
Nov 02 14:58:47.592663 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592488    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istiod-ca-cert" pod "c244ec7c-9329-4f67-bb8d-cb67da5ec034" (UID: "c244ec7c-9329-4f67-bb8d-cb67da5ec034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:47.592663 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592533    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-kube-api-access-fmpkv\" (UniqueName: \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-kube-api-access-fmpkv\") pod \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") "
Nov 02 14:58:47.592663 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592591    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-certs\") pod \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\" (UID: \"c244ec7c-9329-4f67-bb8d-cb67da5ec034\") "
Nov 02 14:58:47.592817 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.592762    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-credential-socket" pod "c244ec7c-9329-4f67-bb8d-cb67da5ec034" (UID: "c244ec7c-9329-4f67-bb8d-cb67da5ec034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:47.593033 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.593010    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-certs" pod "c244ec7c-9329-4f67-bb8d-cb67da5ec034" (UID: "c244ec7c-9329-4f67-bb8d-cb67da5ec034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:47.593141 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.593086    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:47.593141 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.593108    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:47.593216 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.593199    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-socket" pod "c244ec7c-9329-4f67-bb8d-cb67da5ec034" (UID: "c244ec7c-9329-4f67-bb8d-cb67da5ec034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:47.593279 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.593257    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-data" pod "c244ec7c-9329-4f67-bb8d-cb67da5ec034" (UID: "c244ec7c-9329-4f67-bb8d-cb67da5ec034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:47.595025 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.594994    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-envoy" pod "c244ec7c-9329-4f67-bb8d-cb67da5ec034" (UID: "c244ec7c-9329-4f67-bb8d-cb67da5ec034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:47.595218 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.595195    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-token" pod "c244ec7c-9329-4f67-bb8d-cb67da5ec034" (UID: "c244ec7c-9329-4f67-bb8d-cb67da5ec034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:47.595288 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.595212    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-kube-api-access-fmpkv" pod "c244ec7c-9329-4f67-bb8d-cb67da5ec034" (UID: "c244ec7c-9329-4f67-bb8d-cb67da5ec034"). InnerVolumeSpecName "kube-api-access-fmpkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:47.595335 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.595318    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-podinfo" pod "c244ec7c-9329-4f67-bb8d-cb67da5ec034" (UID: "c244ec7c-9329-4f67-bb8d-cb67da5ec034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:58:47.694386 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.694340    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:47.694386 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.694381    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:47.694386 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.694393    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:47.694662 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.694404    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c244ec7c-9329-4f67-bb8d-cb67da5ec034-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:47.694662 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.694417    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:47.694662 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.694429    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmpkv\" (UniqueName: \"kubernetes.io/projected/c244ec7c-9329-4f67-bb8d-cb67da5ec034-kube-api-access-fmpkv\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:47.694662 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:47.694443    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c244ec7c-9329-4f67-bb8d-cb67da5ec034-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:48.250013 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:48.249976    6284 generic.go:356] "Generic (PLEG): container finished" podID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerID="c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228" exitCode=0
Nov 02 14:58:48.250229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:48.250030    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"
Nov 02 14:58:48.250229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:48.250058    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" event={"ID":"c244ec7c-9329-4f67-bb8d-cb67da5ec034","Type":"ContainerDied","Data":"c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228"}
Nov 02 14:58:48.250229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:48.250125    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9" event={"ID":"c244ec7c-9329-4f67-bb8d-cb67da5ec034","Type":"ContainerDied","Data":"f0cfcd025a45aa561d57f916fbad901653fc4f9389e30586001eac195002779c"}
Nov 02 14:58:48.250229 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:48.250155    6284 scope.go:122] "RemoveContainer" containerID="c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228"
Nov 02 14:58:48.260129 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:48.260106    6284 scope.go:122] "RemoveContainer" containerID="c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228"
Nov 02 14:58:48.260510 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:58:48.260487    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228\": container with ID starting with c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228 not found: ID does not exist" containerID="c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228"
Nov 02 14:58:48.260574 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:48.260521    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228"} err="failed to get container status \"c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228\": rpc error: code = NotFound desc = could not find container \"c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228\": container with ID starting with c09e5b199b264fc7f9986653bef945cebc4e65db87b30a01ba79076643964228 not found: ID does not exist"
Nov 02 14:58:48.309177 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:48.309047    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"]
Nov 02 14:58:48.318367 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:48.318290    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-8b5p9"]
Nov 02 14:58:49.126901 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:49.126855    6284 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 14:58:49.127354 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:49.126965    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:58:50.171264 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.171011    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" path="/var/lib/kubelet/pods/c244ec7c-9329-4f67-bb8d-cb67da5ec034/volumes"
Nov 02 14:58:50.189061 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.189031    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:50.221625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.221521    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-credential-socket\") pod \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") "
Nov 02 14:58:50.221625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.221585    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-kube-api-access-hxl2l\" (UniqueName: \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-kube-api-access-hxl2l\") pod \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") "
Nov 02 14:58:50.221625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.221608    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-token\" (UniqueName: \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-token\") pod \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") "
Nov 02 14:58:50.221625 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.221671    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-data\") pod \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") "
Nov 02 14:58:50.222114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.221812    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-certs\") pod \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") "
Nov 02 14:58:50.222114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.221882    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istiod-ca-cert\") pod \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") "
Nov 02 14:58:50.222114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.221891    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-credential-socket" pod "025e5b6b-01b4-4e30-b6ee-8f496082cde5" (UID: "025e5b6b-01b4-4e30-b6ee-8f496082cde5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:50.222114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.221970    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-podinfo\") pod \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") "
Nov 02 14:58:50.222114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.222011    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-envoy\") pod \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") "
Nov 02 14:58:50.222114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.222045    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-socket\") pod \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\" (UID: \"025e5b6b-01b4-4e30-b6ee-8f496082cde5\") "
Nov 02 14:58:50.222114 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.222068    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-data" pod "025e5b6b-01b4-4e30-b6ee-8f496082cde5" (UID: "025e5b6b-01b4-4e30-b6ee-8f496082cde5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:50.222504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.222119    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-certs" pod "025e5b6b-01b4-4e30-b6ee-8f496082cde5" (UID: "025e5b6b-01b4-4e30-b6ee-8f496082cde5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:50.222504 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.222436    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-socket" pod "025e5b6b-01b4-4e30-b6ee-8f496082cde5" (UID: "025e5b6b-01b4-4e30-b6ee-8f496082cde5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:50.222617 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.222592    6284 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-data\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.222739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.222614    6284 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.222739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.222629    6284 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-workload-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.222739 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.222644    6284 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-credential-socket\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.222924 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.222345    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istiod-ca-cert" pod "025e5b6b-01b4-4e30-b6ee-8f496082cde5" (UID: "025e5b6b-01b4-4e30-b6ee-8f496082cde5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:50.224319 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.224294    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-envoy" pod "025e5b6b-01b4-4e30-b6ee-8f496082cde5" (UID: "025e5b6b-01b4-4e30-b6ee-8f496082cde5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:50.224319 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.224304    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-token" pod "025e5b6b-01b4-4e30-b6ee-8f496082cde5" (UID: "025e5b6b-01b4-4e30-b6ee-8f496082cde5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:50.224532 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.224512    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-kube-api-access-hxl2l" pod "025e5b6b-01b4-4e30-b6ee-8f496082cde5" (UID: "025e5b6b-01b4-4e30-b6ee-8f496082cde5"). InnerVolumeSpecName "kube-api-access-hxl2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:50.224532 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.224518    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-podinfo" pod "025e5b6b-01b4-4e30-b6ee-8f496082cde5" (UID: "025e5b6b-01b4-4e30-b6ee-8f496082cde5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:58:50.263755 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.263672    6284 generic.go:356] "Generic (PLEG): container finished" podID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerID="4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310" exitCode=0
Nov 02 14:58:50.263755 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.263717    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" event={"ID":"025e5b6b-01b4-4e30-b6ee-8f496082cde5","Type":"ContainerDied","Data":"4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310"}
Nov 02 14:58:50.264069 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.263805    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz" event={"ID":"025e5b6b-01b4-4e30-b6ee-8f496082cde5","Type":"ContainerDied","Data":"df9232b9f6334a9ec6cca20c3b4451105793e7cdbc8e9fc39eb138ec5f334ab8"}
Nov 02 14:58:50.264069 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.263735    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"
Nov 02 14:58:50.264069 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.263840    6284 scope.go:122] "RemoveContainer" containerID="4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310"
Nov 02 14:58:50.277336 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.277311    6284 scope.go:122] "RemoveContainer" containerID="4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310"
Nov 02 14:58:50.277749 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:58:50.277724    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310\": container with ID starting with 4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310 not found: ID does not exist" containerID="4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310"
Nov 02 14:58:50.277849 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.277763    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310"} err="failed to get container status \"4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310\": rpc error: code = NotFound desc = could not find container \"4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310\": container with ID starting with 4479f6dcc2398fd0897947cb146e97a65239ea10a902865f038eb7ef68ee7310 not found: ID does not exist"
Nov 02 14:58:50.311814 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.311769    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"]
Nov 02 14:58:50.315503 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.315460    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-7svwz"]
Nov 02 14:58:50.323498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.323460    6284 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istiod-ca-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.323498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.323494    6284 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-podinfo\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.323498 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.323505    6284 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-envoy\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.323769 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.323514    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxl2l\" (UniqueName: \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-kube-api-access-hxl2l\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.323769 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:50.323523    6284 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025e5b6b-01b4-4e30-b6ee-8f496082cde5-istio-token\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 14:58:52.168661 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:58:52.168622    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" path="/var/lib/kubelet/pods/025e5b6b-01b4-4e30-b6ee-8f496082cde5/volumes"
Nov 02 14:59:01.465488 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:59:01.465407    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zjqv8_d440b792-e072-43f3-9569-50a6b20f6070/kuadrant-console-plugin/0.log"
Nov 02 14:59:03.337150 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:59:03.337098    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-57f54_88caf1c7-f64a-437c-a169-e6d629102ffc/manager/0.log"
Nov 02 14:59:03.528208 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:59:03.528172    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dpkzz_9926c06f-cb52-4ff4-92fe-468bafff10bb/manager/0.log"
Nov 02 14:59:03.727885 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:59:03.727795    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j74b9_3f60483b-1b89-404c-8782-8efca76bb8d1/manager/0.log"
Nov 02 14:59:04.339171 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:59:04.339131    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f746cf4d4-kg8n2_11b3febd-e5d8-45ac-bc8c-944b268d2b2f/manager/0.log"
Nov 02 14:59:13.895869 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:59:13.895830    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 14:59:14.965217 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:59:14.965179    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zqrhs_cc86d196-682a-4302-9892-6bbbf509bff0/manager/0.log"
Nov 02 14:59:15.478658 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:59:15.478616    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-96kq9_3a19987d-f187-4012-8627-c56ea413d6e7/proxy/0.log"
Nov 02 14:59:29.311443 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:59:29.311358    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:59:29.323800 ip-10-0-131-98 kubenswrapper[6284]: I1102 14:59:29.323765    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 14:59:43.895483 ip-10-0-131-98 kubenswrapper[6284]: E1102 14:59:43.895438    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:00:00.556000 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.555952    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-kg8n2"]
Nov 02 15:00:00.556602 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.556199    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2" podUID="11b3febd-e5d8-45ac-bc8c-944b268d2b2f" containerName="manager" containerID="cri-o://abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b" gracePeriod=30
Nov 02 15:00:00.816598 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.816527    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 15:00:00.867130 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.867084    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-webhook-cert\") pod \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") "
Nov 02 15:00:00.867130 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.867122    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-metrics-certs\") pod \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") "
Nov 02 15:00:00.867402 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.867283    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-kube-api-access-xdwrb\" (UniqueName: \"kubernetes.io/projected/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-kube-api-access-xdwrb\") pod \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\" (UID: \"11b3febd-e5d8-45ac-bc8c-944b268d2b2f\") "
Nov 02 15:00:00.869689 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.869653    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-metrics-certs" pod "11b3febd-e5d8-45ac-bc8c-944b268d2b2f" (UID: "11b3febd-e5d8-45ac-bc8c-944b268d2b2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:00:00.870074 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.870041    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-webhook-cert" pod "11b3febd-e5d8-45ac-bc8c-944b268d2b2f" (UID: "11b3febd-e5d8-45ac-bc8c-944b268d2b2f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:00:00.870264 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.870231    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-kube-api-access-xdwrb" pod "11b3febd-e5d8-45ac-bc8c-944b268d2b2f" (UID: "11b3febd-e5d8-45ac-bc8c-944b268d2b2f"). InnerVolumeSpecName "kube-api-access-xdwrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:00.968622 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.968573    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdwrb\" (UniqueName: \"kubernetes.io/projected/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-kube-api-access-xdwrb\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 15:00:00.968622 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.968614    6284 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-webhook-cert\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 15:00:00.968622 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:00.968624    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11b3febd-e5d8-45ac-bc8c-944b268d2b2f-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 15:00:01.662603 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:01.662561    6284 generic.go:356] "Generic (PLEG): container finished" podID="11b3febd-e5d8-45ac-bc8c-944b268d2b2f" containerID="abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b" exitCode=0
Nov 02 15:00:01.663159 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:01.662615    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2"
Nov 02 15:00:01.663159 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:01.662649    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2" event={"ID":"11b3febd-e5d8-45ac-bc8c-944b268d2b2f","Type":"ContainerDied","Data":"abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b"}
Nov 02 15:00:01.663159 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:01.662710    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f746cf4d4-kg8n2" event={"ID":"11b3febd-e5d8-45ac-bc8c-944b268d2b2f","Type":"ContainerDied","Data":"cd2f120705a66e60688becb4f3dfe0de70c1c982acb171f89b8b686203575956"}
Nov 02 15:00:01.663159 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:01.662733    6284 scope.go:122] "RemoveContainer" containerID="abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b"
Nov 02 15:00:01.675236 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:01.675214    6284 scope.go:122] "RemoveContainer" containerID="abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b"
Nov 02 15:00:01.675631 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:00:01.675611    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b\": container with ID starting with abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b not found: ID does not exist" containerID="abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b"
Nov 02 15:00:01.675723 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:01.675642    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b"} err="failed to get container status \"abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b\": rpc error: code = NotFound desc = could not find container \"abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b\": container with ID starting with abe6d12a1ffd3e9dcba8fab9db0c0db85b6b176e79b9bcbb063dcdefaa3b703b not found: ID does not exist"
Nov 02 15:00:01.707214 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:01.707176    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-kg8n2"]
Nov 02 15:00:01.717088 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:01.717044    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f746cf4d4-kg8n2"]
Nov 02 15:00:02.171257 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:00:02.171214    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b3febd-e5d8-45ac-bc8c-944b268d2b2f" path="/var/lib/kubelet/pods/11b3febd-e5d8-45ac-bc8c-944b268d2b2f/volumes"
Nov 02 15:00:13.894503 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:00:13.894460    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:00:43.894757 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:00:43.894715    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:01:13.894822 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:01:13.894775    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:01:43.893517 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:01:43.893472    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:02:10.183106 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:02:10.183045    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:02:10.183868 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:02:10.183831    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:02:13.895791 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:02:13.895745    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:02:43.896180 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:02:43.896095    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:03:13.944698 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:03:13.944654    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:03:43.896647 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:03:43.896601    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:04:13.894904 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:04:13.894856    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:04:29.350747 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:04:29.350717    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 15:04:29.366116 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:04:29.366081    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 15:04:43.894799 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:04:43.894752    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:05:13.895275 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:05:13.895226    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:05:43.895006 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:05:43.894904    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:05:55.193149 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:05:55.193082    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:05:55.194018 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:05:55.193919    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:13.895692 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:06:13.895649    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:06:43.895476 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:06:43.895432    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:07:13.895789 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:07:13.895733    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:07:43.895848 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:07:43.895801    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:08:13.895096 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:08:13.895053    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:08:43.895051 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:08:43.894967    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:09:13.895151 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:09:13.895109    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:09:29.399214 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:09:29.399177    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 15:09:29.408050 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:09:29.408021    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 15:09:43.894887 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:09:43.894838    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:10:10.182101 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:10:10.182054    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:10:10.182808 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:10:10.182768    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:13.896227 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:10:13.896183    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:10:43.895438 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:10:43.895393    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:11:13.895307 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:11:13.895266    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:11:43.895009 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:11:43.894893    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:12:13.902869 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:12:13.902817    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:12:43.895930 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:12:43.895880    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:13:06.141186 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:13:06.141129    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lfj9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:06.141630 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:13:06.141257    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" podUID="17e32694-fbfc-4adc-aef1-7936996fe885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:06.331164 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:13:06.331117    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dpxqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:06.331402 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:13:06.331187    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" podUID="578e528e-aff8-416c-94b5-9bf6cc01a4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:13.894896 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:13:13.894853    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:13:43.896254 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:13:43.896209    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:14:13.895734 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:14:13.895690    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:14:29.442115 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:14:29.442031    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 15:14:29.450393 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:14:29.450363    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 15:14:43.895818 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:14:43.895772    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:15:13.894835 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:15:13.894796    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:15:43.897346 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:15:43.897302    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:16:13.895499 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:16:13.895434    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:16:43.896080 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:16:43.896035    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:17:13.894219 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:17:13.894171    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:17:43.895823 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:17:43.895733    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:18:13.894914 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:18:13.894873    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:18:43.895641 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:18:43.895597    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:19:13.895594 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:19:13.895548    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:19:17.176183 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176145    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"]
Nov 02 15:19:17.176599 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176546    6284 cpu_manager.go:414] "containerMap: removing container" podUID="9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1" containerName="maas-api"
Nov 02 15:19:17.176599 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176556    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1" containerName="maas-api"
Nov 02 15:19:17.176599 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176581    6284 cpu_manager.go:414] "containerMap: removing container" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy"
Nov 02 15:19:17.176599 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176586    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy"
Nov 02 15:19:17.176599 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176595    6284 cpu_manager.go:414] "containerMap: removing container" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main"
Nov 02 15:19:17.176599 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176601    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main"
Nov 02 15:19:17.176800 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176612    6284 cpu_manager.go:414] "containerMap: removing container" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy"
Nov 02 15:19:17.176800 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176619    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy"
Nov 02 15:19:17.176800 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176629    6284 cpu_manager.go:414] "containerMap: removing container" podUID="11b3febd-e5d8-45ac-bc8c-944b268d2b2f" containerName="manager"
Nov 02 15:19:17.176800 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176634    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b3febd-e5d8-45ac-bc8c-944b268d2b2f" containerName="manager"
Nov 02 15:19:17.176800 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176704    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="025e5b6b-01b4-4e30-b6ee-8f496082cde5" containerName="istio-proxy"
Nov 02 15:19:17.176800 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176713    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c53689-f9e1-42f5-b613-27c0f318f778" containerName="main"
Nov 02 15:19:17.176800 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176721    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="c244ec7c-9329-4f67-bb8d-cb67da5ec034" containerName="istio-proxy"
Nov 02 15:19:17.176800 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176727    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b3febd-e5d8-45ac-bc8c-944b268d2b2f" containerName="manager"
Nov 02 15:19:17.176800 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.176736    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="9acc4f9b-b2cf-4f19-bdcb-9eda4bad9af1" containerName="maas-api"
Nov 02 15:19:17.186383 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.186337    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.218194 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.218148    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"]
Nov 02 15:19:17.225452 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.225403    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/896fc1e0-f4c0-48d6-9809-e5facbceb558-maas-api-tls\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.225644 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.225549    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfrhj\" (UniqueName: \"kubernetes.io/projected/896fc1e0-f4c0-48d6-9809-e5facbceb558-kube-api-access-qfrhj\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.225644 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.225593    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/896fc1e0-f4c0-48d6-9809-e5facbceb558-openshift-service-ca\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.225722 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.225642    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896fc1e0-f4c0-48d6-9809-e5facbceb558-metrics-certs\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.326472 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.326425    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/896fc1e0-f4c0-48d6-9809-e5facbceb558-maas-api-tls\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.326680 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.326501    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfrhj\" (UniqueName: \"kubernetes.io/projected/896fc1e0-f4c0-48d6-9809-e5facbceb558-kube-api-access-qfrhj\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.326680 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.326533    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/896fc1e0-f4c0-48d6-9809-e5facbceb558-openshift-service-ca\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.326680 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.326569    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896fc1e0-f4c0-48d6-9809-e5facbceb558-metrics-certs\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.327287 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.327260    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/896fc1e0-f4c0-48d6-9809-e5facbceb558-openshift-service-ca\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.329129 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.329106    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896fc1e0-f4c0-48d6-9809-e5facbceb558-metrics-certs\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.329239 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.329191    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/896fc1e0-f4c0-48d6-9809-e5facbceb558-maas-api-tls\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.347921 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.347878    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfrhj\" (UniqueName: \"kubernetes.io/projected/896fc1e0-f4c0-48d6-9809-e5facbceb558-kube-api-access-qfrhj\") pod \"maas-api-77c79c44b5-wt48m\" (UID: \"896fc1e0-f4c0-48d6-9809-e5facbceb558\") " pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.502689 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.502645    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:17.811398 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.811304    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"]
Nov 02 15:19:17.815140 ip-10-0-131-98 kubenswrapper[6284]: W1102 15:19:17.815092    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896fc1e0_f4c0_48d6_9809_e5facbceb558.slice/crio-4dc136f1959cd9c0cde0c7ee0a434e6ef8a864a826afd19d4c1e0fc667da5935 WatchSource:0}: Error finding container 4dc136f1959cd9c0cde0c7ee0a434e6ef8a864a826afd19d4c1e0fc667da5935: Status 404 returned error can't find the container with id 4dc136f1959cd9c0cde0c7ee0a434e6ef8a864a826afd19d4c1e0fc667da5935
Nov 02 15:19:17.816503 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:17.816482    6284 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:19:18.781766 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:18.781727    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m" event={"ID":"896fc1e0-f4c0-48d6-9809-e5facbceb558","Type":"ContainerStarted","Data":"df6d37f51b34ceac2525d6a959e4088b385094c7b346755359bb224adb3c8eb5"}
Nov 02 15:19:18.782261 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:18.781781    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m" event={"ID":"896fc1e0-f4c0-48d6-9809-e5facbceb558","Type":"ContainerStarted","Data":"4dc136f1959cd9c0cde0c7ee0a434e6ef8a864a826afd19d4c1e0fc667da5935"}
Nov 02 15:19:18.782261 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:18.781906    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:18.821392 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:18.821324    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m" podStartSLOduration=1.506292083 podStartE2EDuration="1.821306535s" totalImagesPullingTime="315.014452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:18.820222409 +0000 UTC m=+5690.779598066" watchObservedRunningTime="2026-11-02 15:19:18.821306535 +0000 UTC m=+5690.780682189"
Nov 02 15:19:24.794397 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:24.794344    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77c79c44b5-wt48m"
Nov 02 15:19:24.868351 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:24.868303    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"]
Nov 02 15:19:24.868604 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:24.868570    6284 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6" podUID="17ae1c66-7df3-41ec-9e6f-94c76428ad59" containerName="maas-api" containerID="cri-o://cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950" gracePeriod=30
Nov 02 15:19:26.427895 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.427858    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 15:19:26.438744 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.438703    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ae1c66-7df3-41ec-9e6f-94c76428ad59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17ae1c66-7df3-41ec-9e6f-94c76428ad59-openshift-service-ca\") pod \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") "
Nov 02 15:19:26.438929 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.438786    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-maas-api-tls\") pod \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") "
Nov 02 15:19:26.438929 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.438849    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-metrics-certs\") pod \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") "
Nov 02 15:19:26.439043 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.439011    6284 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ae1c66-7df3-41ec-9e6f-94c76428ad59-kube-api-access-hpt8s\" (UniqueName: \"kubernetes.io/projected/17ae1c66-7df3-41ec-9e6f-94c76428ad59-kube-api-access-hpt8s\") pod \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\" (UID: \"17ae1c66-7df3-41ec-9e6f-94c76428ad59\") "
Nov 02 15:19:26.439134 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.439101    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ae1c66-7df3-41ec-9e6f-94c76428ad59-openshift-service-ca" pod "17ae1c66-7df3-41ec-9e6f-94c76428ad59" (UID: "17ae1c66-7df3-41ec-9e6f-94c76428ad59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:26.439379 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.439356    6284 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17ae1c66-7df3-41ec-9e6f-94c76428ad59-openshift-service-ca\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 15:19:26.441173 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.441140    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ae1c66-7df3-41ec-9e6f-94c76428ad59-kube-api-access-hpt8s" pod "17ae1c66-7df3-41ec-9e6f-94c76428ad59" (UID: "17ae1c66-7df3-41ec-9e6f-94c76428ad59"). InnerVolumeSpecName "kube-api-access-hpt8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:26.441173 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.441161    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-maas-api-tls" pod "17ae1c66-7df3-41ec-9e6f-94c76428ad59" (UID: "17ae1c66-7df3-41ec-9e6f-94c76428ad59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:19:26.441332 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.441211    6284 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-metrics-certs" pod "17ae1c66-7df3-41ec-9e6f-94c76428ad59" (UID: "17ae1c66-7df3-41ec-9e6f-94c76428ad59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:19:26.539843 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.539808    6284 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-maas-api-tls\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 15:19:26.540107 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.540081    6284 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17ae1c66-7df3-41ec-9e6f-94c76428ad59-metrics-certs\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 15:19:26.540107 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.540103    6284 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpt8s\" (UniqueName: \"kubernetes.io/projected/17ae1c66-7df3-41ec-9e6f-94c76428ad59-kube-api-access-hpt8s\") on node \"ip-10-0-131-98.ec2.internal\" DevicePath \"\""
Nov 02 15:19:26.872799 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.872752    6284 generic.go:356] "Generic (PLEG): container finished" podID="17ae1c66-7df3-41ec-9e6f-94c76428ad59" containerID="cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950" exitCode=0
Nov 02 15:19:26.873030 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.872824    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6" event={"ID":"17ae1c66-7df3-41ec-9e6f-94c76428ad59","Type":"ContainerDied","Data":"cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950"}
Nov 02 15:19:26.873030 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.872899    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6" event={"ID":"17ae1c66-7df3-41ec-9e6f-94c76428ad59","Type":"ContainerDied","Data":"8064595990caec1c377f4d3fb507c213d049805d7c17c34afaa1b1fcb6ec977f"}
Nov 02 15:19:26.873030 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.872899    6284 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"
Nov 02 15:19:26.873030 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.872955    6284 scope.go:122] "RemoveContainer" containerID="cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950"
Nov 02 15:19:26.885244 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.885219    6284 scope.go:122] "RemoveContainer" containerID="cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950"
Nov 02 15:19:26.885617 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:19:26.885569    6284 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950\": container with ID starting with cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950 not found: ID does not exist" containerID="cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950"
Nov 02 15:19:26.885691 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.885602    6284 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950"} err="failed to get container status \"cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950\": rpc error: code = NotFound desc = could not find container \"cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950\": container with ID starting with cd1b7329bfe1cd7b073cf0800c061c07734fd800a62415a844be1a1f6155d950 not found: ID does not exist"
Nov 02 15:19:26.918499 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.918461    6284 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"]
Nov 02 15:19:26.928326 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:26.928286    6284 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-c77ff87df-7xkq6"]
Nov 02 15:19:28.168163 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:28.168084    6284 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ae1c66-7df3-41ec-9e6f-94c76428ad59" path="/var/lib/kubelet/pods/17ae1c66-7df3-41ec-9e6f-94c76428ad59/volumes"
Nov 02 15:19:29.483114 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:29.483079    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 15:19:29.494587 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:19:29.494553    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 15:19:43.895670 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:19:43.895623    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:20:13.894813 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:20:13.894766    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:20:43.896464 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:20:43.896422    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:21:13.895745 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:21:13.895695    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:21:43.895432 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:21:43.895382    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:21:51.330240 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:21:51.330124    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dpxqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:21:51.330804 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:21:51.330241    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dpxqg" podUID="578e528e-aff8-416c-94b5-9bf6cc01a4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:55.195961 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:21:55.192496    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:21:55.195961 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:21:55.192601    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:22:13.897054 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:22:13.897000    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:22:43.942100 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:22:43.942057    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:23:13.944490 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:23:13.944446    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:23:25.190041 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:23:25.189078    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:23:25.190041 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:23:25.189185    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:23:43.901374 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:23:43.901327    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:23:55.187961 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:23:55.187092    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-49rxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:23:55.187961 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:23:55.187221    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-49rxk" podUID="eb79c248-2d1c-42a1-82dd-57d1545d5588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:06.139195 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:06.139145    6284 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lfj9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:24:06.139785 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:06.139239    6284 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lfj9p" podUID="17e32694-fbfc-4adc-aef1-7936996fe885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:13.896375 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:24:13.896247    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:24:29.523233 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:29.523198    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 15:24:29.537720 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:29.537692    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77n2zk_70acb887-3146-4f03-97cc-19904033a3dc/extract/0.log"
Nov 02 15:24:43.895777 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:24:43.895731    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:24:46.994967 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:46.994901    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-57f54_88caf1c7-f64a-437c-a169-e6d629102ffc/manager/0.log"
Nov 02 15:24:47.222230 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:47.222187    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dpkzz_9926c06f-cb52-4ff4-92fe-468bafff10bb/manager/0.log"
Nov 02 15:24:47.455808 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:47.455720    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j74b9_3f60483b-1b89-404c-8782-8efca76bb8d1/manager/0.log"
Nov 02 15:24:48.198810 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:48.198769    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zqrhs_cc86d196-682a-4302-9892-6bbbf509bff0/manager/0.log"
Nov 02 15:24:48.422332 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:48.422295    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-96kq9_3a19987d-f187-4012-8627-c56ea413d6e7/proxy/0.log"
Nov 02 15:24:49.579884 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:49.579846    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-77c79c44b5-wt48m_896fc1e0-f4c0-48d6-9809-e5facbceb558/maas-api/0.log"
Nov 02 15:24:52.955118 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:52.955011    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-nfrh8_7d03cb08-5ee6-4fb7-b643-53c7b0477bfc/postgres/0.log"
Nov 02 15:24:55.454894 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:55.454859    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zjqv8_d440b792-e072-43f3-9569-50a6b20f6070/kuadrant-console-plugin/0.log"
Nov 02 15:24:57.707044 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:57.706998    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd64f76dd-lv2hm_c3827235-f5e4-44cb-8b10-af6b6d82b14b/kube-auth-proxy/0.log"
Nov 02 15:24:57.894368 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:57.894331    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-49rxk_eb79c248-2d1c-42a1-82dd-57d1545d5588/istio-proxy/0.log"
Nov 02 15:24:58.486855 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:58.486821    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-dpxqg_578e528e-aff8-416c-94b5-9bf6cc01a4cc/istio-proxy/0.log"
Nov 02 15:24:58.884313 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:24:58.884198    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lfj9p_17e32694-fbfc-4adc-aef1-7936996fe885/istio-proxy/0.log"
Nov 02 15:25:13.894515 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:25:13.894470    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:25:38.781221 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:38.781182    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vsx66_570c7233-f4c2-4c16-b9d9-67baa3fa7b96/global-pull-secret-syncer/0.log"
Nov 02 15:25:39.663905 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:39.663862    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pd8nn_72ca4634-5648-402b-aaef-13f8d4ee2b4b/konnectivity-agent/0.log"
Nov 02 15:25:39.905798 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:39.905756    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-98.ec2.internal_41b5d311172faec6b74eed63c4c5b623/haproxy/0.log"
Nov 02 15:25:42.119462 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:42.119433    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-57f54_88caf1c7-f64a-437c-a169-e6d629102ffc/manager/0.log"
Nov 02 15:25:42.245361 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:42.245322    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dpkzz_9926c06f-cb52-4ff4-92fe-468bafff10bb/manager/0.log"
Nov 02 15:25:42.995387 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:42.995352    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j74b9_3f60483b-1b89-404c-8782-8efca76bb8d1/manager/0.log"
Nov 02 15:25:43.294525 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:43.294418    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zqrhs_cc86d196-682a-4302-9892-6bbbf509bff0/manager/0.log"
Nov 02 15:25:43.335263 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:43.335219    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-96kq9_3a19987d-f187-4012-8627-c56ea413d6e7/proxy/0.log"
Nov 02 15:25:43.895258 ip-10-0-131-98 kubenswrapper[6284]: E1102 15:25:43.895219    6284 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lrzvt" containerName="extractor"
Nov 02 15:25:44.679370 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:44.679342    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-57f54_88caf1c7-f64a-437c-a169-e6d629102ffc/manager/0.log"
Nov 02 15:25:44.803257 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:44.803220    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dpkzz_9926c06f-cb52-4ff4-92fe-468bafff10bb/manager/0.log"
Nov 02 15:25:45.539313 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:45.539268    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j74b9_3f60483b-1b89-404c-8782-8efca76bb8d1/manager/0.log"
Nov 02 15:25:45.882079 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:45.881987    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zqrhs_cc86d196-682a-4302-9892-6bbbf509bff0/manager/0.log"
Nov 02 15:25:45.954800 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:45.954765    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-96kq9_3a19987d-f187-4012-8627-c56ea413d6e7/proxy/0.log"
Nov 02 15:25:46.279156 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:46.279125    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zjqv8_d440b792-e072-43f3-9569-50a6b20f6070/kuadrant-console-plugin/0.log"
Nov 02 15:25:47.239403 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:47.239355    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-57f54_88caf1c7-f64a-437c-a169-e6d629102ffc/manager/0.log"
Nov 02 15:25:47.365103 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:47.365061    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-dpkzz_9926c06f-cb52-4ff4-92fe-468bafff10bb/manager/0.log"
Nov 02 15:25:48.117682 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.117646    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f7554a7-ea8c-401f-853e-35b1dcc01d9f/alertmanager/0.log"
Nov 02 15:25:48.146617 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.146585    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-j74b9_3f60483b-1b89-404c-8782-8efca76bb8d1/manager/0.log"
Nov 02 15:25:48.231016 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.230986    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f7554a7-ea8c-401f-853e-35b1dcc01d9f/config-reloader/0.log"
Nov 02 15:25:48.351889 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.351860    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f7554a7-ea8c-401f-853e-35b1dcc01d9f/kube-rbac-proxy-web/0.log"
Nov 02 15:25:48.368981 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.368800    6284 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"]
Nov 02 15:25:48.369292 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.369279    6284 cpu_manager.go:414] "containerMap: removing container" podUID="17ae1c66-7df3-41ec-9e6f-94c76428ad59" containerName="maas-api"
Nov 02 15:25:48.369344 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.369293    6284 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ae1c66-7df3-41ec-9e6f-94c76428ad59" containerName="maas-api"
Nov 02 15:25:48.369386 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.369364    6284 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ae1c66-7df3-41ec-9e6f-94c76428ad59" containerName="maas-api"
Nov 02 15:25:48.372688 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.372665    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.388253 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.388222    6284 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gnwq9\"/\"default-dockercfg-8v5rb\""
Nov 02 15:25:48.390841 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.390805    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"]
Nov 02 15:25:48.398122 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.398085    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gnwq9\"/\"openshift-service-ca.crt\""
Nov 02 15:25:48.398122 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.398110    6284 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gnwq9\"/\"kube-root-ca.crt\""
Nov 02 15:25:48.471325 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.471290    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f7554a7-ea8c-401f-853e-35b1dcc01d9f/kube-rbac-proxy/0.log"
Nov 02 15:25:48.471498 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.471423    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zqrhs_cc86d196-682a-4302-9892-6bbbf509bff0/manager/0.log"
Nov 02 15:25:48.486029 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.485930    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-podres\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.486232 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.486109    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-sys\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.486232 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.486158    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-proc\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.486323 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.486268    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-lib-modules\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.486323 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.486306    6284 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88wkg\" (UniqueName: \"kubernetes.io/projected/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-kube-api-access-88wkg\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.512884 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.512850    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-96kq9_3a19987d-f187-4012-8627-c56ea413d6e7/proxy/0.log"
Nov 02 15:25:48.584204 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.584168    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f7554a7-ea8c-401f-853e-35b1dcc01d9f/kube-rbac-proxy-metric/0.log"
Nov 02 15:25:48.587466 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.587433    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-lib-modules\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.587623 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.587473    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wkg\" (UniqueName: \"kubernetes.io/projected/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-kube-api-access-88wkg\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.587623 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.587507    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-podres\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.587623 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.587592    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-lib-modules\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.587623 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.587598    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-podres\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.587807 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.587664    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-sys\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.587807 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.587715    6284 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-proc\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.587807 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.587757    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-sys\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.587807 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.587796    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-proc\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.608256 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.608227    6284 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88wkg\" (UniqueName: \"kubernetes.io/projected/a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289-kube-api-access-88wkg\") pod \"perf-node-gather-daemonset-gv5kh\" (UID: \"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.687001 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.686872    6284 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:48.697486 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.697454    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f7554a7-ea8c-401f-853e-35b1dcc01d9f/prom-label-proxy/0.log"
Nov 02 15:25:48.823293 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.823258    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6f7554a7-ea8c-401f-853e-35b1dcc01d9f/init-config-reloader/0.log"
Nov 02 15:25:48.984276 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:48.984236    6284 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"]
Nov 02 15:25:48.988141 ip-10-0-131-98 kubenswrapper[6284]: W1102 15:25:48.988109    6284 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda5edd3e8_b0ef_43fc_b5fd_f1c925c6a289.slice/crio-afb6888de94dbe0eacf0ec33f264a0a30c64ef3001d03fe8583b0aabfcf47a7e WatchSource:0}: Error finding container afb6888de94dbe0eacf0ec33f264a0a30c64ef3001d03fe8583b0aabfcf47a7e: Status 404 returned error can't find the container with id afb6888de94dbe0eacf0ec33f264a0a30c64ef3001d03fe8583b0aabfcf47a7e
Nov 02 15:25:49.071445 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:49.071409    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k6t4c_b0175cfb-9276-42ed-bff8-f359a5727eb7/kube-state-metrics/0.log"
Nov 02 15:25:49.193605 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:49.193575    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k6t4c_b0175cfb-9276-42ed-bff8-f359a5727eb7/kube-rbac-proxy-main/0.log"
Nov 02 15:25:49.308907 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:49.308831    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k6t4c_b0175cfb-9276-42ed-bff8-f359a5727eb7/kube-rbac-proxy-self/0.log"
Nov 02 15:25:49.342139 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:49.342102    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh" event={"ID":"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289","Type":"ContainerStarted","Data":"0c13884c2121619b2d82481fd06ea77a5455e586286da95b1b2cef8bc18312b2"}
Nov 02 15:25:49.342335 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:49.342163    6284 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh" event={"ID":"a5edd3e8-b0ef-43fc-b5fd-f1c925c6a289","Type":"ContainerStarted","Data":"afb6888de94dbe0eacf0ec33f264a0a30c64ef3001d03fe8583b0aabfcf47a7e"}
Nov 02 15:25:49.342335 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:49.342186    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:49.377825 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:49.377759    6284 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh" podStartSLOduration=1.377739734 podStartE2EDuration="1.377739734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 15:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:49.377015058 +0000 UTC m=+6081.336390668" watchObservedRunningTime="2026-11-02 15:25:49.377739734 +0000 UTC m=+6081.337115341"
Nov 02 15:25:49.427181 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:49.427152    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-566f4c7487-mbcfx_7dc2b96a-b979-4de1-a69b-ce1c566518c6/metrics-server/0.log"
Nov 02 15:25:50.014521 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:50.014493    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jgpx4_fad8eb6b-fa80-41f6-b5ff-51c39bba446c/node-exporter/0.log"
Nov 02 15:25:50.127242 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:50.127213    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jgpx4_fad8eb6b-fa80-41f6-b5ff-51c39bba446c/kube-rbac-proxy/0.log"
Nov 02 15:25:50.239751 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:50.239718    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jgpx4_fad8eb6b-fa80-41f6-b5ff-51c39bba446c/init-textfile/0.log"
Nov 02 15:25:50.736849 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:50.736808    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dphbb_380def42-3dee-46c1-83b8-52910cc74811/kube-rbac-proxy-main/0.log"
Nov 02 15:25:50.850772 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:50.850734    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dphbb_380def42-3dee-46c1-83b8-52910cc74811/kube-rbac-proxy-self/0.log"
Nov 02 15:25:50.965685 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:50.965655    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dphbb_380def42-3dee-46c1-83b8-52910cc74811/openshift-state-metrics/0.log"
Nov 02 15:25:51.096707 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:51.096619    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cf8d1f5-65fc-4bdc-98d7-9b3732996772/prometheus/0.log"
Nov 02 15:25:51.215909 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:51.215877    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cf8d1f5-65fc-4bdc-98d7-9b3732996772/config-reloader/0.log"
Nov 02 15:25:51.334067 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:51.334041    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cf8d1f5-65fc-4bdc-98d7-9b3732996772/thanos-sidecar/0.log"
Nov 02 15:25:51.446502 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:51.446414    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cf8d1f5-65fc-4bdc-98d7-9b3732996772/kube-rbac-proxy-web/0.log"
Nov 02 15:25:51.558864 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:51.558827    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cf8d1f5-65fc-4bdc-98d7-9b3732996772/kube-rbac-proxy/0.log"
Nov 02 15:25:51.621216 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:51.621180    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd64f76dd-lv2hm_c3827235-f5e4-44cb-8b10-af6b6d82b14b/kube-auth-proxy/0.log"
Nov 02 15:25:51.671066 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:51.671034    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cf8d1f5-65fc-4bdc-98d7-9b3732996772/kube-rbac-proxy-thanos/0.log"
Nov 02 15:25:51.690847 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:51.690809    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-49rxk_eb79c248-2d1c-42a1-82dd-57d1545d5588/istio-proxy/0.log"
Nov 02 15:25:51.783587 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:51.783562    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5cf8d1f5-65fc-4bdc-98d7-9b3732996772/init-config-reloader/0.log"
Nov 02 15:25:51.888668 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:51.888636    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-dpxqg_578e528e-aff8-416c-94b5-9bf6cc01a4cc/istio-proxy/0.log"
Nov 02 15:25:52.021790 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:52.021738    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lfj9p_17e32694-fbfc-4adc-aef1-7936996fe885/istio-proxy/0.log"
Nov 02 15:25:55.358366 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:55.358334    6284 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-gv5kh"
Nov 02 15:25:56.814797 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:25:56.814763    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zjqv8_d440b792-e072-43f3-9569-50a6b20f6070/kuadrant-console-plugin/0.log"
Nov 02 15:26:00.602212 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:26:00.602168    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lq2nc_8e76a365-0c53-4369-b49e-b19a6611a391/dns/0.log"
Nov 02 15:26:00.715670 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:26:00.715638    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lq2nc_8e76a365-0c53-4369-b49e-b19a6611a391/kube-rbac-proxy/0.log"
Nov 02 15:26:00.831645 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:26:00.831605    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5jbwx_35941a8e-d549-436f-b3e1-0be3ef2cc3a7/dns-node-resolver/0.log"
Nov 02 15:26:02.591120 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:26:02.591084    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gkg2v_99d52cb6-4109-40d5-8fc7-fb6679a03d60/node-ca/0.log"
Nov 02 15:26:02.725901 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:26:02.725872    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd64f76dd-lv2hm_c3827235-f5e4-44cb-8b10-af6b6d82b14b/kube-auth-proxy/0.log"
Nov 02 15:26:02.810316 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:26:02.810286    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-49rxk_eb79c248-2d1c-42a1-82dd-57d1545d5588/istio-proxy/0.log"
Nov 02 15:26:03.006416 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:26:03.006383    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-dpxqg_578e528e-aff8-416c-94b5-9bf6cc01a4cc/istio-proxy/0.log"
Nov 02 15:26:03.136112 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:26:03.136080    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lfj9p_17e32694-fbfc-4adc-aef1-7936996fe885/istio-proxy/0.log"
Nov 02 15:26:06.507357 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:26:06.507326    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6fd64f76dd-lv2hm_c3827235-f5e4-44cb-8b10-af6b6d82b14b/kube-auth-proxy/0.log"
Nov 02 15:26:06.633427 ip-10-0-131-98 kubenswrapper[6284]: I1102 15:26:06.633394    6284 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-49rxk_eb79c248-2d1c-42a1-82dd-57d1545d5588/istio-proxy/0.log"